Mid Server requests stuck in processing in ECC Queue


Description

Mid Server requests stuck in processing in ECC Queue. 

If you see the mid server is up, validated, and running but the ECC queue is stuck in processing state. And the agent logs show the error similar to the below


2021-05-14 09:36:38 (559) ECCSender.1 WARNING *** WARNING *** Method failed: (https://XXXXX.service-now.com/ecc_queue.do?SOAP&displayvalue=all&redirectSupported=true)HTTP/1.1 500 Internal Server Error with code: 500
2021-05-14 09:36:38 (606) ECCSender.1 WARNING *** WARNING *** Error executing business rule 'XXXX_XXXX'
2021-05-14 09:36:38 (606) ECCSender.1 WARNING *** WARNING *** RemoteGlideRecord failed to send data to https://XXXXX.service-now.com/ with (Error executing business rule 'XXXX-XXXX')
2021-05-14 09:36:38 (606) ECCSender.1 SEVERE *** ERROR *** Error executing business rule 'XXXX-XXXX'
com.service_now.mid.MIDRemoteGlideRecord$RetryableException: Error executing business rule 'XXXX-XXXX'



Release or Environment

Any

Instructions

1. Add "mid.log.level" config paramter with a value as 'debug' in the mid server config parameters.

2. Grab the agent logs.

3. Upon investigating the logs if you find the error similar to the below:

2021-05-14 00:00:54 (369) Worker-Standard:Command-51f712ae1b883010eae23197cc4bcb94 Enqueuing: D:\MidServer1\agent\work\monitors\ECCSender\output_2\ecc_queue.51f712ae1b883010eae23197cc4bcb94.xml
2021-05-14 00:00:54 (371) Worker-Standard:Command-51f712ae1b883010eae23197cc4bcb94 Worker completed: Command source: INC2080130-vpn time: 0:00:15.026

2021-05-14 09:36:38 (559) ECCSender.1 WARNING *** WARNING *** Method failed: (https://XXXXX.service-now.com/ecc_queue.do?SOAP&displayvalue=all&redirectSupported=true)HTTP/1.1 500 Internal Server Error with code: 500
2021-05-14 09:36:38 (606) ECCSender.1 WARNING *** WARNING *** Error executing business rule 'XXXX_XXXX'
2021-05-14 09:36:38 (606) ECCSender.1 WARNING *** WARNING *** RemoteGlideRecord failed to send data to https://XXXXX.service-now.com/ with (Error executing business rule 'XXXX-XXXX')
2021-05-14 09:36:38 (606) ECCSender.1 SEVERE *** ERROR *** Error executing business rule 'XXXX-XXXX'
com.service_now.mid.MIDRemoteGlideRecord$RetryableException: Error executing business rule 'XXXX-XXXX'

2021-05-14 00:09:26 (155) ECCSender.1 WARNING *** WARNING *** MIDRemoteGlideRecord.insert failed, retrying in 10 seconds


4. If you notice the above error closely the worker has completed the job but it is failing at the ECCSender. 

5. Check for any custom Business rules on the ECC queue table. 

6. ECCSender is responsible for inserting an ECC queue input for each xml file.

7. In this case, there is a before insert BR on the ecc_queue table where customer wrote a logic to update incident table.

8. Since the BR is not async, the transaction is held until all the logic is finished, but the transaction erred out and the error is sent back to ECC Sender so ECC Sender doesn’t know that it sent the      response and it will retry sending, and will always fail.

9. Beacuse of this the ECC queue outputs will always get stuck in processing state.

10. In order to resolve this issue, they need to make the BR to run as Async instead of before or after insert. This way the ECCSender thread is not blocked and it will just do its job of inserting an ecc queue input into the ecc_queue table.