Remotely deleting ECC Queue inputs from the MID Server ECCSender folders - Support and Troubleshooting

Remotely deleting ECC Queue inputs from the MID Server ECCSender folders Issue Note: This KB is only for situations where a Caller on a support case doesn't have immediate and easy access to the hosts the MID Servers are installed on.

Once the MID Server platform has executed any probe, for a job from any feature, it saves the results/data to a temporary file on the MID Server disk. This file is an XML representation of the ecc_input record that will shortly appear in the instance. A separate thread called "ECCSender.1" runs every second, and uses the instance SOAP Table API to insert batches of records, doing the highest priority and oldest record first.

There are several scenarios where you might want to clear some records from that queue:

ECCSender keeps retrying and failing to send a record to the instance, infinitely , due to known problems with the XML Parser used: PRB1759604/KB1644334 MID Server performance issue from ECCSender when 0 size xml files in ECCSender folders (fixed in Xanadu) PRB1792071/KB1695722 Mid server ECCsender fails to process XML files and continuously reties, if the file is truncated, or contains invalid/control/unprintable characters (fixed in Yokohama) Huge amounts of Data from Import Set JDBC Data Sources, or LDAP Imports, are blocking the results for other probes being returned to the instance . see KB0813378 JDBC/LDAP Data Source Imports can use a lot of MID Server Disk space Symptoms The most likely symptom is a very long response time for MID Server jobs, where excessive wait times often cause jobs to be timed out or cancelled from the instance-side. Discovery schedules have a max runtime. JRESTMessageV2/SOAPMessageV2 when executing synchronously default to a 30 second timeout. JDBC Data sources also have a 5 minute default timeout set.

The ECC Queue is fundamentally a queue and long wait times can be expected, so should be taken into account when implementing features and integrations, and their MID Server clusters.

Bad practice RESTMessageV2/SOAPMessageV2 jobs in the instance could be long running, blocking threads, causing semaphore/worker thread exhaustion, while waiting for ecc_queue inputs. That's probably the worst case scenario for a backlogged ECCSender queue. See: KB0694711 Outbound REST Web Services RESTMessageV2 and SOAPMessageV2 execute() vs executeAsync() KB0716391 Best practices for handling scriptable outbound REST and SOAP messages on the Now Platform

Release This can be an issue for any Instance version, and feature doing integrations via the MID Server platform

Resolution Warning: This method simply deletes the queues ECC Queue Inputs. That means the job/probe/integration will now never complete, and will need to be cancelled, and then re-run later.

That might be fine for e.g. a regularly scheduled full import, such as for LDAP, or for a Service Graph Connector, or Discovery schedules, where all the data will be brought in again on the next ruin, but could cause data loss or updates to be missed for real-time synchronisation types of integrations. So be aware of what it is you are deleting before deleting them.

This first version of the article is giving Windows MID Server examples, but the ideas are the same for a Linux one, just with the equivalent commands. dir/ls, type/cat, etc.

Table of Contents Creating a Command probe Specific commands to run in the probe List all XML from all ECCSender folders List the contents of a file Delete a file Creating a Command probe A Command probe will run a command from the Command Prompt (Windows) or shell (Linux) of the MID Server host.

Open the ECC Queue list, and click New to open a new blank record form Fill in the following: Agent: mid.server. Topic: Command (capital C, it's case sensitive) Name: (leave blank) Queue: Output State: Ready Sequence: clear this value, leaving it blank Payload: (In this example, the command is "dir /s work\monitors\ECCSender\ecc_queue.*" in the "name" parameter) dir /s work\monitors\ECCSender\ecc_queue.* "/> Right click the header, and click Save Refresh the Queues related list until the response/input ECC Queue record appears Open the response record and view the Payload field for the stdout and stderr response. Use the orange XML icon to format the XML value, which may make it easier to read, although that has the downside of removing charriage returns. Notes:

The Name parameter is being used rather than the name field to avoid any command length limitations.

The Priority is deliberately being set to Interactive, assuming that the Standard and/or Expedited queues are already blocked. This probe would also be blocked if run at Standard priority.

Queues related list:

Response ECC Queue input record, and XML

Specific commands to run in the probe The folder the file is in will be equivalent to the Priority, except for multi-record inputs from JDBC and LDAP probes, which will be in output_s.

output_2 - Standard Priority output_1 - Expedited Priority output_0 - Interactive Priority output_s - Sequential inputs, from JDBC and LDAP probes The file name will always start with "ecc_queue.", normally followed by the sys_id of the Output record for the job. If you know the Output for the job, you can limit yourself to only the records for that job.

This KB will help you find the Output records in the ecc_queue KB0727132 How to link an ECC Queue record back to a specific Feature or Job

List all XML from all ECCSender folders dir /s work\monitors\ECCSender\ecc_queue.* "/>

Example Response, which includes the timestamp showing how long the file has been there:

List the contents of a file You will usually want to do this to both confirm the file is the one you think it is, or to gather infomation to determine the root cause of a problem, for example which product/integration is it from, XML format issues, unprintable or Unicode characters, truncated file etc.

For this example, lets assume the file is one of the stuck ones, being retried infinitely. In that situation the file name changes each retry , with the number before .xml incrementing each time. We'll use List, with a Wildcard, which would usually be used for listing the contents of multiple files, but this way we find the file without knowing the exact name.

type work\monitors\ECCSender\output_2\ecc_queue.1938229b7650000001.*.xml "/>

Response, in which you can see the file has now been retired 25 times, and is not really an xml file at all. This was a deliberately bad file created for this test.

Delete a file This example uses the same wildcard, in case the retires caused the name to change.

del /Q work\monitors\ECCSender\output_2\ecc_queue.1938229b7650000001.*.xml "/>

If it does delete, you expect no output. If the file wasn't found, you see the error "Could Not Find..." in the response.

You could also delete all records in a folder, although that's probably dangerous without at least backing up the files first, which would need access to the host anyway. See the warning above. Using "copy" first, or "move" instead of delete would be better.

del /Q /S work\monitors\ECCSender\output_s\*.xml "/>