<h2>Events [em_event] are stuck and not processing</h2><br/><div style="overflow-x:auto"><article><div ><h3 >Issue </h3><section><ul style="list-style-position: inside;"><li>It is observed that the records in the Events [em_event] table are stuck and are not processing. When we checked the events table, we can see there are thousands of records stuck in the ready state.<br /><br />https://<Instance_name>.service-now.com/em_event_list.do?sysparm_query=sys_created_onONToday%40javascript%3Ags.beginningOfToday()%40javascript%3Ags.endOfToday()%5Estate%3DReady&sysparm_view=</li></ul></section></div><div ><h3 >Cause</h3><section><ul style="list-style-position: inside;"><li>We checked the sys_trigger table to see the "Event Management - process events" where we observed that the jobs were stuck. <br /><a href="https://<instance_name>.service-now.com/sys_trigger_list.do?sysparm_query=nameLIKEprocess%20events&sysparm_view=">https://<instance_name>.service-now.com/sys_trigger_list.do?sysparm_query=nameLIKEprocess%20events&sysparm_view=</a></li><li>End user tried updating the multi-node property to recreate new jobs but still, the same issue was observed. Upon checking the node logs for one of the nodes, we observed the below error</li></ul> <p style="padding-left: 40px;"><span style="color: #ecf0f1; background-color: #e03e2d;">Actual EM jobs ({44}) and job count ({52}) stored in database are not same, Stopping EM processing jobs till its fixed by Coordinator job</span><br /><br />2023-05-08 01:03:20 (759) worker.4 worker.4 txid=64d0e3e3c3ae Name: Event Management - process events - 1<br />Job Context:<br />#Mon May 08 01:02:50 PDT 2023<br />fcScriptName=in the schedule record<br /><br />Script:<br />var eventProcessor = new SNC.EvtMgmtEventProcessor(); eventProcessor.processWithLimit(0,25);<br />2023-05-08 01:03:20 (764) worker.5 worker.5 txid=28d02b23c3ae DEBUG: Secrets Management: Cache miss for SCOPE:global. Querying repo.<br />2023-05-08 01:03:20 (764) worker.4 worker.4 txid=64d0e3e3c3ae EvtMgmtEventProcessor: Actual EM jobs ({44}) and job count ({52}) stored in database are not same, Stopping EM processing jobs till its fixed by Coordinator job<br />2023-05-08 01:03:20 (764) worker.5 worker.5 txid=28d02b23c3ae DEBUG: Secrets Management: Nothing in repo for SCOPE:global.<br />2023-05-08 01:03:20 (765) worker.5 worker.5 txid=28d02b23c3ae DEBUG: Secrets Management: Cache miss for PACKAGE:ab62e81f1bf533001ead7403dd4bcb52. Querying repo.<br />2023-05-08 01:03:20 (765) worker.5 worker.5 txid=28d02b23c3ae DEBUG: Secrets Management: Nothing in repo for PACKAGE:ab62e81f1bf533001ead7403dd4bcb52.<br />2023-05-08 01:03:20 (765) worker.5 worker.5 txid=28d02b23c3ae DEBUG: Secrets Management: Cache miss for TABLE:sys_flow_secure_data. Querying repo.<br />2023-05-08 01:03:20 (766) worker.4 worker.4 txid=64d0e3e3c3ae Completed: Event Management - process events - 1.8860236783aee11061f1f1c5eeaad387 in 0:00:00.005, next occurrence is 2023-05-08 </p> <ul style="list-style-position: inside;"><li>This means that the job in DB and the job on the UI are not matching. This is because the coordinator job was not in sync/running.</li><li>In the current case, the "Event Management - Coordinator Job" was stuck in a running state on an inactive node thus it we were seeing the error in the logs.<br /><br /><img style="border-style: solid; border-width: 2px;" src="sys_attachment.do?sys_id=3151332f93e6e110f538fb2d6cba104d" alt="Pasted image.png" width="1078" height="352" /></li></ul></section></div><div ><h3 >Resolution</h3><section><ul style="list-style-position: inside;"><li>In order to fix the issue, we have recreated this job by deactivating and activating the job.</li><li>Post that everything started working. All the events are processed now and system is processing new events as well. </li></ul></section></div></article></div>