SMTP Sender schedule job remains in Queued state.


Description

Scheduled jobs named "SMTP Sender" would occasionally remain in "queued" state and not advance to "running". Emails would not be processed and back up.

Release or Environment

Behavior observed in on-prem instance but possible in hosted instances on any version.

Cause

During the normal processing of the "SMTP Sender" job, the schedule type is set to "interval" with a a time window of one minute. The job is in "Ready" state, it is picked up by an active node and the state is changed to "Queued".

The node that picked up the "SMTP Sender" job was an admin node set to "offline" but running so the job state would change to "Queued" but never processed by the node.

Resolution

First, the admin node was first shut down. Then, the state of the "SMTP Sender" job was changed from "Queued" to "Ready". This allowed another "online" node to pick up the job which then changed the state to "Queued" and then "Running". All backed up emails were processed.