Accented characters and strings with spaces used in api calls - Support and Troubleshooting

Accented characters and strings with spaces used in api calls Issue Assuming that a webservice is using words that are passed on in a request towards or from the instance it is usually tested with the basic ascii characters, when used in Europe there is a higher change accented characters are being used and then the webservice starts failing/not retuning results.

Depending on how the request is made/passed the issue can be due to the following RFC:

2.2. URL Character Encoding Issues

URLs are sequences of characters, i.e., letters, digits, and special characters. A URLs may be represented in a variety of ways: e.g., ink on paper, or a sequence of octets in a coded character set. The interpretation of a URL depends only on the identity of the characters used.

In most URL schemes, the sequences of characters in different parts of a URL are used to represent sequences of octets used in Internet protocols. For example, in the ftp scheme, the host name, directory name and file names are such sequences of octets, represented by parts of the URL. Within those parts, an octet may be represented by

the chararacter which has that octet as its code within the US-ASCII [20] coded character set.

In addition, octets may be encoded by a character triplet consisting of the character "%" followed by the two hexadecimal digits (from "0123456789ABCDEF") which forming the hexadecimal value of the octet. (The characters "abcdef" may also be used in hexadecimal encodings.)

Octets must be encoded if they have no corresponding graphic character within the US-ASCII coded character set, if the use of the corresponding character is unsafe, or if the corresponding character is reserved for some other interpretation within the particular URL scheme.

No corresponding graphic US-ASCII:

URLs are written only with the graphic printable characters of the US-ASCII coded character set. The octets 80-FF hexadecimal are not used in US-ASCII, and the octets 00-1F and 7F hexadecimal represent control characters; these must be encoded.

Release all

Cause Assuming that a webservice is using words that are passed on in a request towards or from the instance it is usually tested with the basic ascii characters, when used in Europe there is a higher change accented characters are being used and then the webservice starts failing/not retuning results.

This will aslo affect queries ith strings with spaces and/or reserverd characters that could be part of a URL (?/& etc)

Depending on how the request is made/passed the issue can be due to the following RFC:

2.2. URL Character Encoding Issues

URLs are sequences of characters, i.e., letters, digits, and special characters. A URLs may be represented in a variety of ways: e.g., ink on paper, or a sequence of octets in a coded character set. The interpretation of a URL depends only on the identity of the characters used.

In most URL schemes, the sequences of characters in different parts of a URL are used to represent sequences of octets used in Internet protocols. For example, in the ftp scheme, the host name, directory name and file names are such sequences of octets, represented by parts of the URL. Within those parts, an octet may be represented by

the chararacter which has that octet as its code within the US-ASCII [20] coded character set.

In addition, octets may be encoded by a character triplet consisting of the character "%" followed by the two hexadecimal digits (from "0123456789ABCDEF") which forming the hexadecimal value of the octet. (The characters "abcdef" may also be used in hexadecimal encodings.)

Octets must be encoded if they have no corresponding graphic character within the US-ASCII coded character set, if the use of the corresponding character is unsafe, or if the corresponding character is reserved for some other interpretation within the particular URL scheme.

No corresponding graphic US-ASCII:

URLs are written only with the graphic printable characters of the US-ASCII coded character set. The octets 80-FF hexadecimal are not used in US-ASCII, and the octets 00-1F and 7F hexadecimal represent control characters; these must be encoded.

Assuming you create an api call to the knowledge search and you want to return KB's based on a keyword:

(The script itself is not relevant as the issue is with the URL)

https://instance.service-now.com/api/namespace/search_knowledge_articles/searchfor/Error

this will return the KB's containing the word 'error'

When pasted in a browser this will indeed return the KB's with that searchfor term (error) in it.

However when using the same url but with the word 'felanmäler' (reporting an error in Swedish):

encodeURIComponentfelanmäler

there will be no results.

If you log the search terms in the api call you will see that the term is seen as 'felanmÃ¤ler', and there will be no results.

The cause of this issue is how URL's work. See the above RFC.

The proof of this is when you paste the above URL in a browser, then copy the URL again from the browser and paste it in a text file the URL will look as follows:

https://instance.service-now.com/api/namespace/search_knowledge_articles/searchfor/felanm%C3%A4lercc

Resolution

Ensure the term you search for is encoded for URI's before the request is send and decoded when the request is received in the instance

Encoding (javascript) in

var wordToSearch="felanmäler"

var encWordToSearch= encodeURIComponent(wordToSearch)

var url= "https://instance.service-now.com/api/namespace/search_knowledge_articles/searchfor/"+encWordToSearch

On the instance:

//untested as the parameters in the request may vary

var encWordToSearch =request.pathParams.searchtext;

// note: patParams explanation link below

var WordToSearch = decodeURIComponent(encWordToSearch)

// rest of the code

Related Links Information on request.pathparams: https://developer.servicenow.com/blog.do?p=/post/introduction-to-scripted-rest-apis/

Information on encodeURIComponent: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/encodeURIComponent

Information on decodeURIComponent: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/decodeURIComponent

Other programming languages have there own functions fo decoding/encoding, please refer to the appropriate resources