Mid server ECCsender fails to process XML files and continuously reties, if the file is truncated, or contains invalid/control/unprintable characters - Known Error

Mid server ECCsender fails to process XML files and continuously reties, if the file is truncated, or contains invalid/control/unprintable characters Description Mid server ECCsender fails to process XML files and continuously reties, if the file is truncated, or contains unparsable characters like #31;

Steps to Reproduce

In a working MID Server, paste the 2 test files attached to the problem into agent\work\monitors\ECCSender\output_2 1/ ecc_queue.chr031.xml - this includes "#31;" in a value. Other invalid/unprintable characters are likely to also break this. 2/ ecc_queue.truncated.xml - this is valid XML, up to the point that the file is truncated

You will see they soon start getting renamed (e.g. ecc_queue.chr031.4.xml, ecc_queue.chr031.5.xml...) as they are retried, and will continue to fail to be sent to the instance.

There is no attempt to send to the instance because it seems to fail to read them, and also does not give up and move the records to output_error as you would expect for a failure to send (since PRB1521761 fix in San Diego). MID Server property mid.eccq.sender_max_retries_on_server_error also has no effect.

Agent log will show:

1/

2024-08-02T14:20:29.067+0200 ERROR (ECCSender.1) [SimpleSaxParser$SaxHandler:110] Character reference "&#31" is an invalid XML character. org.xml.sax.SAXParseException; systemId: file:/C:/.../agent/work/monitors/ECCSender/output_2/ecc_queue.chr031.18.xml; lineNumber: 1; columnNumber: 111991; Character reference "&#31" is an invalid XML character. at org.apache.xerces.util.ErrorHandlerWrapper.createSAXParseException(Unknown Source) at org.apache.xerces.util.ErrorHandlerWrapper.fatalError(Unknown Source) at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source) at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source) at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source) at org.apache.xerces.impl.XMLScanner.reportFatalError(Unknown Source) at org.apache.xerces.impl.XMLScanner.scanCharReferenceValue(Unknown Source) at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanCharReference(Unknown Source) at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown Source) at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source) at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source) at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source) at org.apache.xerces.parsers.XMLParser.parse(Unknown Source) at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source) at org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source) at org.apache.xerces.jaxp.SAXParserImpl.parse(Unknown Source) at java.xml/javax.xml.parsers.SAXParser.parse(SAXParser.java:330) at com.service_now.monitor. SimpleSaxParser.parseAsMap(SimpleSaxParser.java:44) at com.service_now.monitor.ECCSenderQueueFile.parseDataUsingFile(ECCSenderQueueFile.java:163) at com.service_now.monitor.ECCSenderCache.loadData(ECCSenderCache.java:451) at com.service_now.monitor.ECCSenderCache.processFile(ECCSenderCache.java:380) at com.service_now.monitor.ECCSenderCache.sendFile(ECCSenderCache.java:342) at com.service_now.monitor.ECCSenderCache.sendFiles(ECCSenderCache.java:296) at com.service_now.monitor.ECCSender.run(ECCSender.java:122) at com.snc.midserver.monitor.internal.MonitorRunner$MonitorTask.execute(MonitorRunner.java:235) at com.snc.midserver.monitor.internal.AMonitorTask.run(AMonitorTask.java:29) at java.base/java.util.TimerThread.mainLoop(Timer.java:566) at java.base/java.util.TimerThread.run(Timer.java:516) 2024-08-02T14:20:29.067+0200 WARN (ECCSender.1) [ECCSenderQueueFile:167] Failed to parse XML using SAX parser: Character reference "&#31" is an invalid XML character.; reverting to using DOM parser 2024-08-02T14:20:29.069+0200 WARN (ECCSender.1) [XMLUtil:527] org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 111991; Character reference "&#31" is an invalid XML character. 2024-08-02T14:20:29.069+0200 ERROR (ECCSender.1) [ECCSenderCache:389] Failure sending file: ecc_queue.chr031.18.xml java.io.IOException: Failed to parse XML using DOM parser, null returned - file: C:\...\agent\work\monitors\ECCSender\output_2\ecc_queue.chr031.18.xml at com.service_now.monitor.ECCSenderQueueFile.parseDataUsingFile(ECCSenderQueueFile.java:182) at com.service_now.monitor.ECCSenderCache.loadData(ECCSenderCache.java:451) at com.service_now.monitor.ECCSenderCache.processFile(ECCSenderCache.java:380) at com.service_now.monitor.ECCSenderCache.sendFile(ECCSenderCache.java:342) at com.service_now.monitor.ECCSenderCache.sendFiles(ECCSenderCache.java:296) at com.service_now.monitor.ECCSender.run(ECCSender.java:122) at com.snc.midserver.monitor.internal.MonitorRunner$MonitorTask.execute(MonitorRunner.java:235) at com.snc.midserver.monitor.internal.AMonitorTask.run(AMonitorTask.java:29) at java.base/java.util.TimerThread.mainLoop(Timer.java:566) at java.base/java.util.TimerThread.run(Timer.java:516) 2024-08-02T14:20:29.070+0200 DEBUG (ECCSender.1) [ECCSenderCache:423] ** sendFile exit: Attempt Count: 19 for C:\...\agent\work\monitors\ECCSender\output_2\ecc_queue.chr031.19.xml 2024-08-02T14:20:29.070+0200 DEBUG (ECCSender.1) [ECCSenderCache:297] ** sending file FAILED!!! C:\...\agent\work\monitors\ECCSender\output_2\ecc_queue.chr031.19.xml 2/

2024-08-02T14:20:29.071+0200 ERROR (ECCSender.1) [SimpleSaxParser$SaxHandler:110] XML document structures must start and end within the same entity. org.xml.sax.SAXParseException; systemId: file:/C:/.../agent/work/monitors/ECCSender/output_2/ecc_queue.truncated.18.xml; lineNumber: 1; columnNumber: 1011; XML document structures must start and end within the same entity. at org.apache.xerces.util.ErrorHandlerWrapper.createSAXParseException(Unknown Source) at org.apache.xerces.util.ErrorHandlerWrapper.fatalError(Unknown Source) at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source) at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source) at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source) at org.apache.xerces.impl.XMLScanner.reportFatalError(Unknown Source) at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.endEntity(Unknown Source) at org.apache.xerces.impl.XMLDocumentScannerImpl.endEntity(Unknown Source) at org.apache.xerces.impl.XMLEntityManager.endEntity(Unknown Source) at org.apache.xerces.impl.XMLEntityScanner.load(Unknown Source) at org.apache.xerces.impl.XMLEntityScanner.scanContent(Unknown Source) at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanContent(Unknown Source) at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown Source) at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source) at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source) at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source) at org.apache.xerces.parsers.XMLParser.parse(Unknown Source) at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source) at org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source) at org.apache.xerces.jaxp.SAXParserImpl.parse(Unknown Source) at java.xml/javax.xml.parsers.SAXParser.parse(SAXParser.java:330) at com.service_now.monitor. SimpleSaxParser.parseAsMap(SimpleSaxParser.java:44) at com.service_now.monitor.ECCSenderQueueFile.parseDataUsingFile(ECCSenderQueueFile.java:163) at com.service_now.monitor.ECCSenderCache.loadData(ECCSenderCache.java:451) at com.service_now.monitor.ECCSenderCache.processFile(ECCSenderCache.java:380) at com.service_now.monitor.ECCSenderCache.sendFile(ECCSenderCache.java:342) at com.service_now.monitor.ECCSenderCache.sendFiles(ECCSenderCache.java:296) at com.service_now.monitor.ECCSender.run(ECCSender.java:122) at com.snc.midserver.monitor.internal.MonitorRunner$MonitorTask.execute(MonitorRunner.java:235) at com.snc.midserver.monitor.internal.AMonitorTask.run(AMonitorTask.java:29) at java.base/java.util.TimerThread.mainLoop(Timer.java:566) at java.base/java.util.TimerThread.run(Timer.java:516) 2024-08-02T14:20:29.071+0200 WARN (ECCSender.1) [ECCSenderQueueFile:167] Failed to parse XML using SAX parser: XML document structures must start and end within the same entity.; reverting to using DOM parser 2024-08-02T14:20:29.072+0200 WARN (ECCSender.1) [XMLUtil:527] org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1011; XML document structures must start and end within the same entity. 2024-08-02T14:20:29.072+0200 ERROR (ECCSender.1) [ECCSenderCache:389] Failure sending file: ecc_queue.truncated.18.xml java.io.IOException: Failed to parse XML using DOM parser, null returned - file: C:\...\agent\work\monitors\ECCSender\output_2\ecc_queue.truncated.18.xml at com.service_now.monitor.ECCSenderQueueFile.parseDataUsingFile(ECCSenderQueueFile.java:182) at com.service_now.monitor.ECCSenderCache.loadData(ECCSenderCache.java:451) at com.service_now.monitor.ECCSenderCache.processFile(ECCSenderCache.java:380) at com.service_now.monitor.ECCSenderCache.sendFile(ECCSenderCache.java:342) at com.service_now.monitor.ECCSenderCache.sendFiles(ECCSenderCache.java:296) at com.service_now.monitor.ECCSender.run(ECCSender.java:122) at com.snc.midserver.monitor.internal.MonitorRunner$MonitorTask.execute(MonitorRunner.java:235) at com.snc.midserver.monitor.internal.AMonitorTask.run(AMonitorTask.java:29) at java.base/java.util.TimerThread.mainLoop(Timer.java:566) at java.base/java.util.TimerThread.run(Timer.java:516) Workaround This problem is currently under review and targeted to be fixed in a future release. Subscribe to this Known Error article to receive notifications when more information will be available.

The XML files can be manually deleted from the ECCSender folders, to prevent the infinite retries. The files with older timestamps will be the problem ones.

Control codes/unprintable characters will need removing from the source data. It would be expected for any XML parser to fail if this kind of character is included in the data. This problem will be fixing the error handling, not the root cause, which is the bad data.

Truncated files are likely to be due to forced shutdowns of the MID Server application while files are being written. If restarting the hosts a mid server is installed on, please give it time to shutdown cleanly.