Backlog of jobs in sys_trigger with State: Ready that are not picked up to be processed though there are available scheduler workers
In New York release, if you note that there is a buildup of jobs in the sys_trigger table which are past the datetime in the next_action field and are not picked up for processing, you may be affected by this PRB.
Some of the symptoms include:
- a large number of jobs/records in sys_trigger table (>100,000)
- a large number of jobs (more than 10,000) that are of the same name (such as ASYNC jobs) and having a lower priority (default priority is 100) and the next action datetime has past
- there are a few long-running jobs on one or more nodes
- there are available scheduler workers on various nodes and they looked to be idle (not processing jobs)
Steps to Reproduce
- Access <instance>.service-now.com/sys_trigger.do and filter for the following:
- Next action at or before Today
- State is Ready
- Trigger type is not On Demand
- There will be a large number of records (>100,000).
- Access <instance>.service-now.com/stats.do and review the "Background Scheduler" section, refresh a few times and note that there are available scheduler workers with "Current job: idle"
As a workaround, please create a table index (composite) as follows:
Table: Schedule Item [sys_trigger]
Fields (Labels): Priority,Next action
Fields (name): priority,next_action
Related Problem: PRB1373305