Event Queue Heartbeat - related to Job 'Event Queue Monitor' and glide.heartbeat event - Support and Troubleshooting

Event Queue Heartbeat - related to Job 'Event Queue Monitor' and glide.heartbeat event Issue .ns-kb-css-body-editor-container { p { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } span { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } h2 { font-size: 24pt; font-family: Lato; color: var(--now-color--text-primary, black); } h3 { font-size: 18pt; font-family: Lato; color: var(--now-color--text-primary, black); } h4 { font-size: 14pt; font-family: Lato; color: var(--now-color--text-primary, black); } a { font-size: 12pt; font-family: Lato; color: var(--now-color--link-primary, #00718F); } a:hover { font-size: 12pt; color: var(--now-color--link-primary, #024F69); } a:target { font-size: 12pt; color: var(--now-color--link-primary, #032D42); } a:visited { font-size: 12pt; color: var(--now-color--link-primary, #00718f); } ul { font-size: 12pt; font-family: Lato; } li { font-size: 12pt; font-family: Lato; } img { display: ; max-width: ; width: ; height: ; } } Event Queue Heartbeat alerts are being triggered, or glide.heartbeat events are not being created or processed within expected intervals.

This generally indicates one of the following conditions:

The Event Queue Monitor scheduled job is not executing properly. Heartbeat events are being created but are not being processed. The scheduler is unable to execute the Event Queue Monitor job due to system load. Event processing is experiencing a backlog. The glide.heartbeat event is a platform generated health check event used to verify that event generation and event processing are functioning correctly. When heartbeat events stop being generated or processed, it may indicate an issue with the scheduler, event processing framework, or cluster health.

Release .ns-kb-css-body-editor-container { p { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } span { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } h2 { font-size: 24pt; font-family: Lato; color: var(--now-color--text-primary, black); } h3 { font-size: 18pt; font-family: Lato; color: var(--now-color--text-primary, black); } h4 { font-size: 14pt; font-family: Lato; color: var(--now-color--text-primary, black); } a { font-size: 12pt; font-family: Lato; color: var(--now-color--link-primary, #00718F); } a:hover { font-size: 12pt; color: var(--now-color--link-primary, #024F69); } a:target { font-size: 12pt; color: var(--now-color--link-primary, #032D42); } a:visited { font-size: 12pt; color: var(--now-color--link-primary, #00718f); } ul { font-size: 12pt; font-family: Lato; } li { font-size: 12pt; font-family: Lato; } img { display: ; max-width: ; width: ; height: ; } } All available releases.

Cause .ns-kb-css-body-editor-container { p { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } span { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } h2 { font-size: 24pt; font-family: Lato; color: var(--now-color--text-primary, black); } h3 { font-size: 18pt; font-family: Lato; color: var(--now-color--text-primary, black); } h4 { font-size: 14pt; font-family: Lato; color: var(--now-color--text-primary, black); } a { font-size: 12pt; font-family: Lato; color: var(--now-color--link-primary, #00718F); } a:hover { font-size: 12pt; color: var(--now-color--link-primary, #024F69); } a:target { font-size: 12pt; color: var(--now-color--link-primary, #032D42); } a:visited { font-size: 12pt; color: var(--now-color--link-primary, #00718f); } ul { font-size: 12pt; font-family: Lato; } li { font-size: 12pt; font-family: Lato; } img { display: ; max-width: ; width: ; height: ; } } Cause 1: Event Queue Monitor Job in Invalid State or Claimed by Invalid Node The Event Queue Monitor scheduled job may be:

Claimed by an invalid or orphaned cluster node. Stuck in a Queued state instead of Ready. Prevented from executing at its expected one minute interval. Cause 2: Scheduler Overload Preventing Event Queue Monitor Execution The scheduler may be overwhelmed with excessive scheduled jobs, commonly ASYNC jobs, preventing the Event Queue Monitor from running.

When this occurs:

New heartbeat events are not generated. Scheduler queues continue to grow. Other scheduled jobs may also be delayed. Cause 3: Event Processing Backlog The Event Queue Monitor job executes successfully and creates heartbeat events, but the Events Processor is unable to process events quickly enough.

When this occurs:

Heartbeat events remain in a queued state. The event backlog grows continuously. Event Queue Heartbeat alerts may be triggered. Cause 4: Node Instability or Cluster Communication Issues The Event Queue Monitor may fail to execute correctly when:

Nodes are repeatedly restarting. Cluster communication is degraded. Job ownership information becomes stale. Scheduled jobs remain assigned to unavailable nodes. Evidence may include:

Frequent node changes in sys_cluster_state . Repeated node startup messages in logs. Scheduled jobs repeatedly changing ownership.

Investigation Step 1: Determine if Event Queue Monitor Is Running Navigate to:

sys_trigger_list.do Filter on:

Name = Event Queue Monitor

Verify the following:

State should be Ready. Trigger Type should be Interval. Interval should be 1 minute. Last Ran should update regularly. System ID should reference a valid cluster node. If the job is missing, open a Support case for assistance restoring the out of box scheduled job.

Step 2: Verify Valid Cluster Nodes Navigate to:

sys_cluster_state Determine the active datacenter using:

stats.do Verify:

The node assigned in the Event Queue Monitor record exists in the active datacenter. The assigned node is active and healthy. If the System ID points to an invalid or orphaned node, proceed to the Resolution section.

Step 3: Verify Events Processor Activity Navigate to:

sys_trigger_list.do Search for:

Events Processor Verify:

State is Ready. Last Ran updates regularly. The job is not stuck in Running or Queued state. If the Events Processor is not executing correctly, heartbeat events may be generated but never processed.

Step 4: Check Event Queue Backlog Navigate to:

sysevent_list.do Filter:

State is not one of Error, Processed, Transferred Sort by oldest first.

Observe the queue over several refreshes:

Event count should decrease over time. New heartbeat events should be generated. Existing heartbeat events should eventually be processed. A continuously growing queue may indicate an event processing bottleneck.

Step 5: Check for Scheduler Overload Navigate to:

sys_trigger_list.do Filter:

Name STARTSWITH ASYNC

Review:

Number of queued ASYNC jobs. Whether the queue continues growing over time. Whether large numbers of identical jobs are being created. If a large backlog exists and continues growing, investigate the source of the workload, such as:

Slow database queries. Long running business rules. Runaway integrations. Excessive background processing. Resolution .ns-kb-css-body-editor-container { p { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } span { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } h2 { font-size: 24pt; font-family: Lato; color: var(--now-color--text-primary, black); } h3 { font-size: 18pt; font-family: Lato; color: var(--now-color--text-primary, black); } h4 { font-size: 14pt; font-family: Lato; color: var(--now-color--text-primary, black); } a { font-size: 12pt; font-family: Lato; color: var(--now-color--link-primary, #00718F); } a:hover { font-size: 12pt; color: var(--now-color--link-primary, #024F69); } a:target { font-size: 12pt; color: var(--now-color--link-primary, #032D42); } a:visited { font-size: 12pt; color: var(--now-color--link-primary, #00718f); } ul { font-size: 12pt; font-family: Lato; } li { font-size: 12pt; font-family: Lato; } img { display: ; max-width: ; width: ; height: ; } } Choose the appropriate resolution based on the identified root cause.

Scenario 1: Event Queue Monitor Claimed by Invalid Node Navigate to:

sys_trigger_list.do Open:

Event Queue Monitor Perform the following:

Set the System ID field to empty, or assign a valid active node. Ensure State is set to Ready. Save the record. The scheduler will automatically select a valid node during the next execution cycle.

Scenario 2: Event Queue Monitor Stuck in Queued State Navigate to:

sys_trigger_list.do Open:

Event Queue Monitor

Verify the current state.

If the state is Queued:

Change the state to Ready. Save the record. Monitor the next execution cycle. The job should execute within approximately one minute.

Scenario 3: Scheduler Overload Caused by Excessive ASYNC Jobs Navigate to:

sys_trigger_list.do Filter:

Name STARTSWITH ASYNC

Investigate the root cause of the workload before taking corrective action.

Option A: Remove Excessive ASYNC Jobs If approved by the customer and confirmed to be safe:

Bulk delete excessive ASYNC jobs. Monitor scheduler recovery. Confirm Event Queue Monitor resumes normal execution. Option B: Temporarily Prevent Non-Critical ASYNC Jobs from Executing If deletion is not acceptable:

Review business impact. Obtain customer approval. Temporarily prevent non-critical ASYNC jobs from executing until the backlog is reduced by disabling the active field. Monitor scheduler recovery. After remediation:

Open the Event Queue Monitor record. Select Execute Now. Verify heartbeat events are being generated and processed. Scenario 4: Orphaned Heartbeat Event If a glide.heartbeat event remains queued for an extended period and is not being processed:

Navigate to Component Status ( sys_status_list.do ). Locate the entry: glide.event_heartbeat.last_id Identify the corresponding heartbeat event in the sysevent table. Confirm the event has remained queued and is not being processed. Delete the orphaned event record. Monitor Event Queue Monitor execution. A new heartbeat event should be generated during the next execution cycle.

Scenario 5: Node Instability or Cluster Issues If nodes are restarting frequently or ownership information appears stale:

Review node health and cluster status. Verify all nodes are active and communicating correctly. Confirm scheduled jobs are assigned to valid nodes. Engage ServiceNow Support if cluster instability persists.

Validation and Monitoring After applying any resolution:

Verify Event Queue Monitor Execution Navigate to:

sys_trigger_list.do Filter:

Event Queue Monitor Confirm the job transitions normally:

Ready → Running → Ready approximately every 60 seconds.

Verify Events Processor Activity Confirm the Events Processor job continues executing regularly and is not stuck in a queued or running state.

Verify Event Queue Health Repeat the event backlog verification described in Investigation Step 4.

Expected results:

Event backlog decreases over time. Heartbeat events are generated regularly. Heartbeat events are processed successfully. No new Event Queue Heartbeat alerts are triggered.