How to detect stuck scheduled jobs that may be stuck on a non-existent node - Support and Troubleshooting

How to detect stuck scheduled jobs that may be stuck on a non-existent node Issue  Description The glide.scheduler thread on an application node will claim jobs from the sys_trigger table to run on that node. Sometimes we may encounter the circumstance where a scheduled job is claimed by a node and then some maintenance occurs and the node is retired or moved. In those cases, the claimed jobs will be in a "limbo" state. This knowledge article demonstrates how one can identify such scheduled jobs.

Procedure Navigate to the System Maintenance > Scripts - Background module, copy the following snippet of javascript into the field labeled " Run script (JavaScript executed on server) ", and click on the Run Script button.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 showScheduledJobsStuckOnMissingNodes(); function showScheduledJobsStuckOnMissingNodes() { var gr = new GlideRecord( 'sys_trigger' ); // Query only jobs that are queued or "running" gr.addQuery( 'state' , 'IN' , '1,2' ); // Filter out those that are not claimed by any node gr.addQuery( 'claimed_by' , '!=' , 'NULL' ); // Filter out those that are claimed by a valid node var gr2 = new GlideRecord( 'sys_cluster_state' ); gr2.query(); while (gr2.next()) { gr.addQuery( 'claimed_by' , '!=' , gr2.system_id); // Filter out any valid system_id values. } gr.query(); // Display a warning for each suspect scheduled jobs while (gr.next()) { gs.print( 'WARNING: sys_trigger.' + gr.sys_id + ' has a state of ' + gr.state + ' and is claimed by node ' + gr.claimed_by) } }

Applicable Versions All