Mid server ECCsender fails to process XML files and continuously reties, if the file is truncated, or contains invalid/control/unprintable characters - Known Error

Mid server ECCsender fails to process XML files and continuously reties, if the file is truncated, or contains invalid/control/unprintable characters Description Mid server ECCsender fails to process XML files and continuously reties, if the file is truncated, or contains unparsable characters like #31;

Steps to Reproduce

In a working MID Server, paste the 2 test files attached to the problem into agent\work\monitors\ECCSender\output_2 1/ ecc_queue.chr031.xml - this includes "#31;" in a value. Other invalid/unprintable characters are likely to also break this. 2/ ecc_queue.truncated.xml - this is valid XML, up to the point that the file is truncated

You will see they soon start getting renamed (e.g. ecc_queue.chr031.4.xml, ecc_queue.chr031.5.xml...) as they are retried, and will continue to fail to be sent to the instance.

There is no attempt to send to the instance because it seems to fail to read them, and also does not give up and move the records to output_error as you would expect for a failure to send (since PRB1521761 fix in San Diego). MID Server property mid.eccq.sender_max_retries_on_server_error also has no effect.

Agent log will show:

1/

2024-08-02T14:20:29.067+0200 ERROR (ECCSender.1) [SimpleSaxParser$SaxHandler:110] Character reference "&#31" is an invalid XML character. org.xml.sax.SAXParseException; systemId: file:/C:/.../agent/work/monitors/ECCSender/output_2/ecc_queue.chr031.18.xml; lineNumber: 1; columnNumber: 111991; Character reference "&#31" is an invalid XML character. at org.apache.xerces.util.ErrorHandlerWrapper.createSAXParseException(Unknown Source) at org.apache.xerces.util.ErrorHandlerWrapper.fatalError(Unknown Source) at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source) at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source) at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source) at org.apache.xerces.impl.XMLScanner.reportFatalError(Unknown Source) at org.apache.xerces.impl.XMLScanner.scanCharReferenceValue(Unknown Source) at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanCharReference(Unknown Source) at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown Source) at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source) at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source) at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source) at org.apache.xerces.parsers.XMLParser.parse(Unknown Source) at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source) at org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source) at org.apache.xerces.jaxp.SAXParserImpl.parse(Unknown Source) at java.xml/javax.xml.parsers.SAXParser.parse(SAXParser.java:330) at com.service_now.monitor. SimpleSaxParser.parseAsMap(SimpleSaxParser.java:44) at com.service_now.monitor.ECCSenderQueueFile.parseDataUsingFile(ECCSenderQueueFile.java:163) at com.service_now.monitor.ECCSenderCache.loadData(ECCSenderCache.java:451) at com.service_now.monitor.ECCSenderCache.processFile(ECCSenderCache.java:380) at com.service_now.monitor.ECCSenderCache.sendFile(ECCSenderCache.java:342) at com.service_now.monitor.ECCSenderCache.sendFiles(ECCSenderCache.java:296) at com.service_now.monitor.ECCSender.run(ECCSender.java:122) at com.snc.midserver.monitor.internal.MonitorRunner$MonitorTask.execute(MonitorRunner.java:235) at com.snc.midserver.monitor.internal.AMonitorTask.run(AMonitorTask.java:29) at java.base/java.util.TimerThread.mainLoop(Timer.java:566) at java.base/java.util.TimerThread.run(Timer.java:516) 2024-08-02T14:20:29.067+0200 WARN (ECCSender.1) [ECCSenderQueueFile:167] Failed to parse XML using SAX parser: Character reference "&#31" is an invalid XML character.; reverting to using DOM parser 2024-08-02T14:20:29.069+0200 WARN (ECCSender.1) [XMLUtil:527] org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 111991; Character reference "&#31" is an invalid XML character. 2024-08-02T14:20:29.069+0200 ERROR (ECCSender.1) [ECCSenderCache:389] Failure sending file: ecc_queue.chr031.18.xml java.io.IOException: Failed to parse XML using DOM parser, null returned - file: C:\...\agent\work\monitors\ECCSender\output_2\ecc_queue.chr031.18.xml at com.service_now.monitor.ECCSenderQueueFile.parseDataUsingFile(ECCSenderQueueFile.java:182) at com.service_now.monitor.ECCSenderCache.loadData(ECCSenderCache.java:451) at com.service_now.monitor.ECCSenderCache.processFile(ECCSenderCache.java:380) at com.service_now.monitor.ECCSenderCache.sendFile(ECCSenderCache.java:342) at com.service_now.monitor.ECCSenderCache.sendFiles(ECCSenderCache.java:296) at com.service_now.monitor.ECCSender.run(ECCSender.java:122) at com.snc.midserver.monitor.internal.MonitorRunner$MonitorTask.execute(MonitorRunner.java:235) at com.snc.midserver.monitor.internal.AMonitorTask.run(AMonitorTask.java:29) at java.base/java.util.TimerThread.mainLoop(Timer.java:566) at java.base/java.util.TimerThread.run(Timer.java:516) 2024-08-02T14:20:29.070+0200 DEBUG (ECCSender.1) [ECCSenderCache:423] ** sendFile exit: Attempt Count: 19 for C:\...\agent\work\monitors\ECCSender\output_2\ecc_queue.chr031.19.xml 2024-08-02T14:20:29.070+0200 DEBUG (ECCSender.1) [ECCSenderCache:297] ** sending file FAILED!!! C:\...\agent\work\monitors\ECCSender\output_2\ecc_queue.chr031.19.xml 2/

2024-08-02T14:20:29.071+0200 ERROR (ECCSender.1) [SimpleSaxParser$SaxHandler:110] XML document structures must start and end within the same entity. org.xml.sax.SAXParseException; systemId: file:/C:/.../agent/work/monitors/ECCSender/output_2/ecc_queue.truncated.18.xml; lineNumber: 1; columnNumber: 1011; XML document structures must start and end within the same entity. at org.apache.xerces.util.ErrorHandlerWrapper.createSAXParseException(Unknown Source) at org.apache.xerces.util.ErrorHandlerWrapper.fatalError(Unknown Source) at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source) at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source) at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source) at org.apache.xerces.impl.XMLScanner.reportFatalError(Unknown Source) at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.endEntity(Unknown Source) at org.apache.xerces.impl.XMLDocumentScannerImpl.endEntity(Unknown Source) at org.apache.xerces.impl.XMLEntityManager.endEntity(Unknown Source) at org.apache.xerces.impl.XMLEntityScanner.load(Unknown Source) at org.apache.xerces.impl.XMLEntityScanner.scanContent(Unknown Source) at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanContent(Unknown Source) at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown Source) at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source) at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source) at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source) at org.apache.xerces.parsers.XMLParser.parse(Unknown Source) at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source) at org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source) at org.apache.xerces.jaxp.SAXParserImpl.parse(Unknown Source) at java.xml/javax.xml.parsers.SAXParser.parse(SAXParser.java:330) at com.service_now.monitor. SimpleSaxParser.parseAsMap(SimpleSaxParser.java:44) at com.service_now.monitor.ECCSenderQueueFile.parseDataUsingFile(ECCSenderQueueFile.java:163) at com.service_now.monitor.ECCSenderCache.loadData(ECCSenderCache.java:451) at com.service_now.monitor.ECCSenderCache.processFile(ECCSenderCache.java:380) at com.service_now.monitor.ECCSenderCache.sendFile(ECCSenderCache.java:342) at com.service_now.monitor.ECCSenderCache.sendFiles(ECCSenderCache.java:296) at com.service_now.monitor.ECCSender.run(ECCSender.java:122) at com.snc.midserver.monitor.internal.MonitorRunner$MonitorTask.execute(MonitorRunner.java:235) at com.snc.midserver.monitor.internal.AMonitorTask.run(AMonitorTask.java:29) at java.base/java.util.TimerThread.mainLoop(Timer.java:566) at java.base/java.util.TimerThread.run(Timer.java:516) 2024-08-02T14:20:29.071+0200 WARN (ECCSender.1) [ECCSenderQueueFile:167] Failed to parse XML using SAX parser: XML document structures must start and end within the same entity.; reverting to using DOM parser 2024-08-02T14:20:29.072+0200 WARN (ECCSender.1) [XMLUtil:527] org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1011; XML document structures must start and end within the same entity. 2024-08-02T14:20:29.072+0200 ERROR (ECCSender.1) [ECCSenderCache:389] Failure sending file: ecc_queue.truncated.18.xml java.io.IOException: Failed to parse XML using DOM parser, null returned - file: C:\...\agent\work\monitors\ECCSender\output_2\ecc_queue.truncated.18.xml at com.service_now.monitor.ECCSenderQueueFile.parseDataUsingFile(ECCSenderQueueFile.java:182) at com.service_now.monitor.ECCSenderCache.loadData(ECCSenderCache.java:451) at com.service_now.monitor.ECCSenderCache.processFile(ECCSenderCache.java:380) at com.service_now.monitor.ECCSenderCache.sendFile(ECCSenderCache.java:342) at com.service_now.monitor.ECCSenderCache.sendFiles(ECCSenderCache.java:296) at com.service_now.monitor.ECCSender.run(ECCSender.java:122) at com.snc.midserver.monitor.internal.MonitorRunner$MonitorTask.execute(MonitorRunner.java:235) at com.snc.midserver.monitor.internal.AMonitorTask.run(AMonitorTask.java:29) at java.base/java.util.TimerThread.mainLoop(Timer.java:566) at java.base/java.util.TimerThread.run(Timer.java:516) Workaround This problem is currently under review and targeted to be fixed in a future release. Subscribe to this Known Error article to receive notifications when more information will be available.

There is no workaround to prevent this happening, however the XML files can be manually moved out of the ECCSender folders to prevent the infinite retries. The fix for this error handling problem will be to move the bad records to the ...ECCSender\output_error folder, so we suggest that's what you do manually too.

If you don't have direct access to the MID Server, and it is up and still working, albeit slowly, manually inserted "Command" topic probe outputs into the ecc_queue could be used to create the error folder then move those records. e.g. mkdir work\monitors\ECCSender\output_error move work\monitors\ECCSender\output_s\ecc_queue.19344ad43de0000001.* work\monitors\ECCSender\output_error

The files with older timestamps, and large numbers in the name just before the .xml, will be the problem ones. Inspect the contents first to understand which features jobs these were results from. The kb article KB0727132 How to link an ECC Queue record back to a specific Feature or Job will help you do that.

For import sets or LDAP imports, where the bad characters are in the source data, those Control codes/unprintable characters will need removing from the source data. It would be expected for any XML parser to fail if this kind of character is included in the data. This problem will be fixing the error handling, not the root cause , which is the bad data.

Truncated files are likely to be due to forced shutdowns of the MID Server application while files are being written. If restarting the hosts a mid server is installed on, please give it time to shutdown cleanly.