MID Server will run the same probe twice, if it completes during a MID Server shutdown, when the result is prevented from being written to the ECCSender folder - Known Error

MID Server will run the same probe twice, if it completes during a MID Server shutdown, when the result is prevented from being written to the ECCSender folder Description When a MID Server starts to shut down or restart, it will give probes some time to finish running, however the ECCSender thread, which writes the outputs to the ECCSender folder, is stopped before those finish, and that prevents the results being written to the temporary xml files when the probes end. That means when the MID fully shuts down, the xml file in ECCSender doesn't exist, and the Output record in the ECC Queue table remains in Processing state, and that combination means when the MID Server starts up again the ECCSenderCache code believes the job was never finished, and sets it back to Ready state to run it again.

Symptoms could include duplicate records being created in a 3rd party system, due to push integrations running twice for the same record in servicenow. Other symptoms would depend on the integration being effected.

Steps to Reproduce

This would be tricky to reproduce but not impossible.

The following is what was observed in a customer MID Server:

This was a SOAPProbe thread starting, although it could be any topic/probe:

2026-03-02T13:07:50.803+0100 INFO (Worker-Expedited:SOAPProbe- 19b4637e87537610c1ef32e83cbb359f ) [AWorker:184] Worker starting: SOAPProbe source: ... This is where the MID started to stop, 5 seconds into that probe running:

2026-03-02T13:07:55.331+0100 INFO (WrapperListener_stop_runner) [Main:389] Stopping MID server And it had to stop, because the OS was stopping the service, externally, because the server was restarting for some patching. From wrapper.log:

2026/03/02 13:07:55 | Stopping the ServiceNow MID Server_xxx service... 2026/03/02 13:08:00 | Waiting to stop... 2026/03/02 13:08:05 | Waiting to stop... 2026/03/02 13:08:11 | Waiting to stop... 2026/03/02 13:08:17 | Waiting to stop... 2026/03/02 13:08:22 | Waiting to stop... 2026/03/02 13:08:23 | &lt;-- Wrapper Stopped 2026/03/02 13:08:24 | ServiceNow MID Server_xxx service stopped. that took about 30 seconds to stop. This is that same 30s from the agent log:

It starts to kill all the currently running threads, including SCCSender immediately:

2026-03-02T13:07:56.533+0100 INFO (MIDServer) [Instance:948] interrupting thread ECCSender.1 2026-03-02T13:07:56.594+0100 INFO (MIDServer) [ServiceManager:58] Shutting down services... 2026-03-02T13:07:56.596+0100 INFO (MIDServer) [ServiceManager:61] Stopping service: PriorityThreadPoolProvider [120] 2026-03-02T13:07:56.597+0100 INFO (MIDServer) [PriorityThreadPoolProvider:367] Shutting down ThreadPool-Expedited ... Here is the SOAPProbe completing. but we _don't_ see the expected "Enqueuing: C:\...\ECCSender\output_1\ecc_queue.19b4637e87537610c1ef32e83cbb359f.xml", confirming the result was never written to the ECCSender folders. If it had, there would be no problem, as it would be seen by the ECCSender Cache code on the next startup.

2026-03-02T13:08:08.578+0100 INFO (Worker-Expedited:SOAPProbe- 19b4637e87537610c1ef32e83cbb359f ) [AWorker:208] Worker completed : SOAPProbe source: ... It was 12 seconds later that any remaining Worker-Expedited threads would have been killed.

2026-03-02T13:08:20.492+0100 INFO (MIDServer) [PriorityThreadPoolProvider:376] ThreadPool-Expedited terminated 2026-03-02T13:08:21.673+0100 INFO (MIDServer) [Bootstrapper:60] the ServiceNow MID Server is now terminated Here is the MID Server confirming there is nothing in the ECCSender folders, and setting the output back to Ready:

2026-03-02T13:09:11.399+0100 INFO (StartupSequencer) [ECCSenderCache:215] ECC Sender Cache initialization: step 1 of 5: setting up cache directories... 2026-03-02T13:09:11.401+0100 INFO (StartupSequencer) [ECCSenderCache:222] ECC Sender Cache initialization: step 2 of 5: building list of sysids from each file in cache directories... 2026-03-02T13:09:11.403+0100 INFO (StartupSequencer) [QueueProcessingTracker:97] Initializing state cache with 0 sys_ids from ECCSender work output directory files. 2026-03-02T13:09:11.517+0100 INFO (StartupSequencer) [MIDServerInfoRemote:281] Getting 0 ECC queue output record sys_ids for agent: ... 2026-03-02T13:09:11.570+0100 INFO (StartupSequencer) [MIDServerInfoRemote:281] Getting 23 ECC queue output record sys_ids for agent: ... 2026-03-02T13:09:11.629+0100 INFO (StartupSequencer) [ECCSenderCache:277] ECC Sender Cache initialization: step 3 of 5: for all ECC Queue output records assigned to this MID Server which are in Processing state and do not have input files on disk: change state to Ready... and runs it a second time:

2026-03-02T13:09:29.017+0100 INFO (Worker-Expedited:SOAPProbe-19b4637e87537610c1ef32e83cbb359f) [AWorker:184] Worker starting: SOAPProbe source: ... 2026-03-02T13:09:30.304+0100 INFO (Worker-Expedited:SOAPProbe-19b4637e87537610c1ef32e83cbb359f) [ECCSender:242] Enqueuing : C:\...\agent\work\monitors\ECCSender\output_1\ecc_queue. 19b4637e87537610c1ef32e83cbb359f .xml 2026-03-02T13:09:30.306+0100 INFO (Worker-Expedited:SOAPProbe-19b4637e87537610c1ef32e83cbb359f) [AWorker:208] Worker completed : SOAPProbe source: ... time: 0:00:01.284 That's how it is possible for a job to complete, and make a successful request to the 3rd party system, but ECCSender folder is empty, and ecc_queue output is still Processing, so will run again on startup.

Workaround This problem is currently under review and targeted to be fixed in a future release. Subscribe to this Known Error article to receive notifications when more information will be available.

A workaround is available, which is to ensure nothing is still runnig when the MID Server is stopped, using the Pause function:

Ideally set things up so that new jobs are not being sent to the MID Server in the first place. That would help avoid instance-side timeouts, while the MID Server is going to be down. Set the MID Server to Paused state, using the related link on the mid server form. Watch the ECC Queue outputs in Processing state. Any already executing in MID Server Worker threads will complete, and turn to Processed. If any outputs remain in Processing state, then this will be because they are queued in the MID Server memory, but not started yet. Those will not be run in Paused state, and be lost from the memory on shutdown. Those outputs will remain in Processing state inthe instance, and be automaticlly set to Ready on the next startup so that they can be run. Within 10 minutes, a 'queue.stats' ecc_queue input will appear, which will allow you to confirm nothing is still running in a Worker thread. Alternatively, grabbing the logs and checking the thread dump in wrapper.log will also allow you to confirm nothing is still running in a worker thread. Then shut down the MID Server service, and allow it to cleanly shut down, which should not take more than 30 seconds. The benefit of stopping the service manually is that it gets up to 5 minutes to cleanly shut down. If the OS stops the service as part of a server shutdown, the JVM gets a SIGKILL in 30 seconds only, with no further cleanup. Then do the host restart. Once it is back up, the service will automatically start again. Then Resume the MID Server once it is back up, using the related link on the mid server form. Related Problem: PRB2011363