MID Server ECC queue blocked by the position of the "row" within the XML payload landing between buffers and splitting causing org.xml.sax.SAXParseException; invalid byte 2 of 4-byte UTF-8 sequence - Known Error

MID Server ECC queue blocked by the position of the "row" within the XML payload landing between buffers and splitting causing org.xml.sax.SAXParseException; invalid byte 2 of 4-byte UTF-8 sequence Description The most likely cause of this issue is that at some point in the past the instance made an outbound SOAP call via the MID server, and the XML data received in response to that SOAP call was not able to be parsed. This could be either due to syntax errors in the XML or to character encoding errors in the XML content. It could also be the position of the "row" within the XML payload landing between buffers and splitting the valid UTF8 string

An Invalid byte 2 of 4-byte UTF-8 sequence error due to the unicode in the payload can prevent an ecc_queue input being inserted to the instance. The MID Server's ECCSender thread can get blocked and leave the MID Server effectively Down, as although new jobs will be picked up and run, none of the results will be returned to the instance.

Steps to Reproduce

To test this requires a response payload containing 4-byte unicode, which is difficult to cause on demand. The problem ticket has a real ecc sender xml file attached that does reproduce the issue.

The MID Server agent log will usually just show status 500 for the ECCSender SOAP request to the ecc_queue.

07/13/21 09:34:13 (995) ECCSender.1 WARNING *** WARNING *** Method failed: (https:// .service-now.com/ecc_queue.do?SOAP&displayvalue=all&redirectSupported=true)HTTP/1.1 500 Internal Server Error with code: 500 07/13/21 09:34:13 (995) ECCSender.1 WARNING *** WARNING *** Unable to parse SOAP document 07/13/21 09:34:13 (995) ECCSender.1 WARNING *** WARNING *** RemoteGlideRecord failed to send data to https:// .service-now.com/ with (Unable to parse SOAP document) 07/13/21 09:34:13 (995) ECCSender.1 WARNING *** WARNING *** MIDRemoteGlideRecord.insert failed, retrying in 22 seconds The instance app node logs will be needed to confirm this specific cause.

2021-05-04 23:59:24 (284) API_INT-thread-4 A4BC187F1BFB6C90742E9825BB4BCB11 txid=09dc9cbb1b3f WARNING *** WARNING *** org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 368604; Invalid byte 2 of 4-byte UTF-8 sequence. E.G - The analysis of "ecc_queue.178a46cfcaf0000001.56.xml" showed that the "row" within the file that is triggering this warning is: None SPO 2021-04-02 12:46:58.0 DATA DATA GROUP#0 Отчеты TME Ural DATA DATA DATA DATA ExistingExternalUserSharingOnly Unlocked 6018

If the import ran again successfully, the issue is most likely the position of the "row" within the XML payload landing between buffers and splitting the valid UTF8 string. e.g. "Отчеты TME Ural" (full_name attribute). After testing it was found that adding or removing 1 character anywhere before this string allows it to process successfully.

Workaround This is fixed in Tokyo by PRB1544057 RemoteGlideRecord insert with binary data fails with 500 error due to Xerces library bug.

In the MID Server configuration file "config.xml" add the configuration parameter mentioned below and restart the MID Server.

Note: Adding the above configuration, might remove some data from XML which it cannot parse.

Additional Observation Note: Under the ECCSender check for the output folders and check for any files. Please take the back-up of these below files and delete these files and restart the mid server to validate the LDAP connection. This workaround has fixed the issue when the above errors were seen in ECC.

/ agent / work / monitors / ECCSender / Output_0/1/2