Agent Client Collector Monitoring - How is a check result published or ignored? - Support and Troubleshooting

Agent Client Collector Monitoring - How is a check result published or ignored? .ns-kb-css-body-editor-container { p { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } span { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } h2 { font-size: 24pt; font-family: Lato; color: var(--now-color--text-primary, black); } h3 { font-size: 18pt; font-family: Lato; color: var(--now-color--text-primary, black); } h4 { font-size: 14pt; font-family: Lato; color: var(--now-color--text-primary, black); } a { font-size: 12pt; font-family: Lato; color: var(--now-color--link-primary, #00718F); } a:hover { font-size: 12pt; color: var(--now-color--link-primary, #024F69); } a:target { font-size: 12pt; color: var(--now-color--link-primary, #032D42); } a:visited { font-size: 12pt; color: var(--now-color--link-primary, #00718f); } ul { font-size: 12pt; font-family: Lato; } li { font-size: 12pt; font-family: Lato; } img { display: ; max-width: ; width: ; height: ; } } Contents Introduction When is a check result published? Command Changed Muting Threshold Repetition Threshold Examples Introduction Agent Client Collector check results are processed based on the defined settings in the "Check Type", which is referenced in the check definition.

Check types may include events, metrics, or discovery type checks.

The Check Type can be used to select a MID Script to process the results, or it can be used to define an instance side script for processing the results.

https://docs.servicenow.com/csh?topicname=acc-api-check-type.html&version=latest

Agent Client Collector Monitoring event checks provide functionality to produce em_event records based on check output. As the end goal of these event checks is to create em_event records, it is more effective to write directly to the em_event table than to put the check result into the ecc_queue for instance-side processing.

As such, the default behaviour for Agent Client Collector Monitoring event checks is to use a MID Script and pre-process the check output into an em_event and then insert it directly to the em_event table using a REST API.

Event checks often run on short intervals - such as 60 seconds - it is common that the result will be the same between most of the check executions. Alternatively, given that the event checks may measure something that frequently changes (like CPU usage or Disk Space usage), it may not be appropriate to send an event immediately without waiting to confirm that the issue is persistent.

When is a check result published?

To avoid flooding the em_event table with useless events, the agent will decide if it should publish or ignore the check result before passing it to the MID.

This may lead to an agent log entry that states "Check result is being ignored".

[INFO] [agent] check result is being ignored... This is not an error, but rather done to ensure the data passed to the instance is useful and not excessive.

However, when reviewing check executions to confirm the expected behaviour, the log entry alone does not provide much information.

When the log level is debug, it will also write another log entry that states why the check result was or was not ignored:

[DEBUG] [agent] shouldPublishCheckResult: commandChanged = false, repetitionReached = false, mutingThresholdReached = false

As visible here, there are 3 logical conditions evaluated to determine if the check result should be published or ignored:

1. Has the "command" on the check changed since the last execution? 2. Has the "muting threshold" been met? 3. Has the "repetition threshold" been met?

This is calculated by the agent, using information from the check instance and from the connected MID's properties. The information is stored in agent memory, and so if the agent is restarted the values will all be reset.

The following configurations are used in the decision to publish or ignore the check result:

"Check Type" "Event Status Change Threshold" "Event Status Repair Threshold" "Command" MID Property " sn_agent.agent.time_to_resend_unchanged_non_info_status_sec " The check definition form defines some of these configurations:

https://docs.servicenow.com/csh?topicname=check-definition-form.html&version=latest

Check Type Select the type of check. The available monitoring options are: Events Metrics Command Enter the command that the Agent Client Collector executes. Note: When the Command Auto Generation check box is selected, this field is automatically populated with the prefix and flags of the active parameters specified in the Parameters tab. The field contains parameters taken from one or more of the following:

Simple template: { {.labels.params_field_name}} : Takes values from entries in the Parameters tab. For example, a string of { {.labels.params_warning}} means that the value of the warning parameter is used (provided it has been configured on the Parameters tab). Monitored CI: { {.labels.params_ci_field_name}} : Takes values automatically from the monitored CI, where field_name is replaced with the exact name of the required field in the CI table. Event status change threshold The number of consecutive times that a check's response status must happen before a new event is sent. For example, if this value is set to 5, a check whose response status changes from OK to Error generates a new event with an Error status only after the fifth consecutive occurrence of the status change.

Default value: 5 (except for AWS event policies)

Event status repair threshold The number of consecutive times that a check's response status must improve to close the previous event. For example, if this value is set to 3, a check whose response status changes from Error to OK closes the previous event and generates a new event with an OK status only after the third consecutive occurrence of the status change.

Default value: 1 (except for AWS event policies)

The MID Server property " sn_agent.agent.time_to_resend_unchanged_non_info_status_sec " defines the amount of time that must elapse before a check result is re-published when it has not changed and is not status 0. This property should be added to the ecc_agent_property table - or to the Properties related list on the MID Server form. From there, it is synchronized to the Agent upon agent connection to the MID. If not defined, it has a default value of 86400 - i.e., 1 day.

Note: These properties are only evaluated when the "MID Script" in the agent's check type is "MonitorEventParser" - otherwise the check result is always published.

On first execution, a blank "event check status" is generated - containing the following properties that are updated on each execution:

LastPublishedStatus: -1, LastStatus: -1, LastTimeSent: (now), RepeatCounter: 0, StatusRepaired: false, LastPublishedCommand: "", Note - this contains an empty "command", and a timestamp of "now" (i.e., when the check is first run, which is usually shortly after agent restart or due to policy publishing/recalculation)

"LastPublishedStatus" is the status when the check was last published.

"LastStatus" means the previous status produced by the check, whether it was sent or ignored

After this, the event check status is updated and stored in memory each time the check executes.

First - the check result itself is received from the check execution:

checkResult ={ "client": " ", "agent_id": " ", "check": { "command": "check-cpu.rb { {if .labels.params_critical}} -c { {.labels.params_critical}} { {end}} { {if .labels.params_warning}} -w { {.labels.params_warning}} { {end}}", "name": "policy: Linux OS Events, check:os.linux.check-system-cpu", "interval": 60, "timeout": 60, "output": "Check CPU Total OK: The total CPU utilization is 34.17% |* user=33.67 nice=0.0 system=0.5 idle=65.83 iowait=0.0 irq=0.0 softirq=0.0 steal=0.0 guest=0.0 guest_nice=0.0*| \n", "status": "0", "ci_id": " ", "mid_script": "MonitorEventParser", "check_instance_id": " ", "check_type_id": " ", "params_last_updated": "2024-06-20 18:43:57", "metric_name": "os.linux.metrics-system-cpu", "params_background": "false", "params_critical": "50", "params_warning": "40", "disable_token_replacement": "false", "exec_mode": "shell" }, "timestamp": 1720603940 } And then, this is compared to the previous execution's results.

The values are updated, and then it repeats on the next execution.

Command Changed As "Command" is empty when the check is initialised, and is set upon output of first execution - this is always true for first execution of a check and therefore the result is always published on first execution.

On subsequent executions, this will only republish if the command has changed, such as if a change was made to the policy.

Muting Threshold Muting threshold is based on the "time to repeat" - this is configured per MID rather than per agent/per check.

It is configured with this MID property ( ecc_agent_property ): sn_agent.agent.time_to_resend_unchanged_non_info_status_sec

If not set, it has a default of 86400 seconds (1 day).

The "LastTimeSent" is populated with a (now) timestamp every time a check is published.

The first execution always publishes, so it always has a "LastTimeSent" value.

Muting Threshold is only evaluated if current status is not 0 - i.e., it will not continue to send "OK" events repeatedly even if the repeat time period has passed. Then, it compares "LastTimeSent" + the value of the "sn_agent.agent.time_to_resend_unchanged_non_info_status_sec" property. If that time is greater than (now), it publishes the check result.

This means if an event check that runs every minute and always returns the same status, it will not send the same output every minute - it will only do it one time per day, or if the status changes.

The only way to change this is changing this mid property, which can be a very useful troubleshooting tool but the recommendation is to return to a reasonable value once testing is complete.

Note: Be very careful with this property, it can flood the instance with events if the value is too low, or if it is too high then the check will rarely (or never) re-send which may mean the opportunity to re-open incorrectly closed alerts is missed.

Repetition Threshold This check compares "RepeatCounter" to the configured "Event Status Change Threshold" and "Repair Threshold".

"RepeatCounter" is the number of times in a row that the same status has been returned.

"Event Status Change Threshold" refers to the number of times the new status must repeat when going "up" in severity before we consider publishing the result.

"Repair Threshold" refers to the number of times the new status must repeat when going "down" in severity before we consider publishing the result.

This is determined by comparing current "Status" to "LastPublishedStatus":

- if current "Status" is higher than "LastPublishedStatus", use "Event Status Change Threshold"

- if current status is lower, use "Repair Threshold"

- if it's the same, don't consider either at all.

The "RepeatCounter" is then used to track the number of repetitions to see if we have reached the threshold. It iterates up by 1 every time, and resets every time the LastStatus doesnt match the current status

Note: this compares current status to "LastStatus" not to "LastPublishedStatus" when resetting.

Examples These examples will focus on a CPU Usage Check, and will cover the repetition and muting thresholds.

CPU usage is a value that can fluctuate wildly over even short periods of time, short spikes to high usage are not usually considered a concern but sustained period of high cpu usage may indicate a hung process or may cause further downstream issues if an app server is experiencing cpu exhaustion.

Scenario :

Check CPU usage and return result based on current CPU usage

Interval: 2 minutes

Event Status Change Threshold: 5 Repair Threshold: 3

Thresholds: Warning - 80% Critical - 95%

In this scenario, the first execution gives status 0 (OK), and then repeats 0 before the tests happen, and the tests happen in order.

This means that both "LastStatus" and "LastPublishedStatus" are both 0 at the start.

Test 1 : Status Increases and remains at new higher Status

CPU usage is 85%, and remains at this level for at least 10 minutes

Check status: 1 (warning)

Event Check Status:

"LastStatus" = 0 "LastPublishedStatus" = 0 "Status" = 1 "RepeatCounter" = 1 Evaluation: "Status" (1) > "LastPublishedStatus" (0) therefore it will evaluate the repeat counter against the "Event Status Change Threshold". "RepeatCounter" (1) < Event Status Change Threshold (5), therefore the check result is ignored

The agent log will show:

[INFO] [agent] check result is being ignored... After a 2 minute interval, the next check execution runs and returns the same status: 1

Event Check Status:

"LastStatus" = 1 "LastPublishedStatus" = 0 "Status" = 1 "RepeatCounter" = 2 Evaluation: "Status" (1) > "LastPublishedStatus" (0) therefore we continue to evaluate the repeat counter against the "Event Status Change Threshold". "RepeatCounter" (2) < Event Status Change Threshold (5), therefore the check result is ignored

The agent log will show:

[INFO] [agent] check result is being ignored... This check then repeats on a 2 minute interval with the same result each time, until the RepeatCounter is 5

Event Check Status:

"LastStatus" = 1 "LastPublishedStatus" = 0 "Status" = 1 "RepeatCounter" = 5 Evaluation: "Status" (1) > "LastPublishedStatus" (0) therefore we continue to evaluate the repeat counter against the "Event Status Change Threshold". "RepeatCounter" (5) == Event Status Change Threshold (5), therefore the check result is published (this was after 5 repetitions)

The agent log will show:

[DEBUG] [agent] shouldPublishCheckResult: commandChanged = false, repetitionReached = true, mutingThresholdReached = false Test 2 : Status increases then returns to original status

CPU Usage starts at 85%, goes to 99% for one check execution then returns to 85%

First Check status: 2 (critical)

Event Check Status:

"LastStatus" = 1 "LastPublishedStatus" = 1 "Status" = 2 "RepeatCounter" = 1 Evaluation: "Status" (2) > "LastPublishedStatus" (1) therefore we evaluate the repeat counter against the "Event Status Change Threshold". "RepeatCounter" (1) < Event Status Change Threshold (5), therefore the check result is ignored (it would publish after 5 repetitions)

The agent log will show:

[INFO] [agent] check result is being ignored...

After 2 minute interval, the CPU Usage has returned to 85%

Check status: 1 (warning)

Event Check Status:

"LastStatus" = 2 "LastPublishedStatus" = 1 "Status" = 1 "RepeatCounter" = 1 Evaluation: "Status" (1) == "LastPublishedStatus" (1) therefore it will not evaluate the repeat counter against anything.

The agent log will show:

[INFO] [agent] check result is being ignored... Test 3 : Status decreases - check result is repaired

CPU Usage goes down to 40%, and remains at 40%

Check status: 0 (OK)

Event Check Status:

"LastStatus" = 1 "LastPublishedStatus" = 1 "Status" = 0 "RepeatCounter" = 1 Evaluation: "Status" (0) > "LastPublishedStatus" (1) therefore we evaluate the repeat counter against the "Repair Threshold". "RepeatCounter" (1) < Repair Threshold (3), therefore the check result is ignored (it would publish after 3 repetitions)

The agent log will show:

[INFO] [agent] check result is being ignored...

Following 2 more 2 minute intervals, the same result has now been repeated two more times for a total RepeatCounter of 3:

Event Check Status:

"LastStatus" = 0 "LastPublishedStatus" = 1 "Status" = 0 "RepeatCounter" = 3 Evaluation: "Status" (0) > "LastPublishedStatus" (1) therefore we evaluate the repeat counter against the "Repair Threshold". "RepeatCounter" (3) == Repair Threshold (3), therefore the check result is published

The agent log will show:

[DEBUG] [agent] shouldPublishCheckResult: commandChanged = false, repetitionReached = true, mutingThresholdReached = false

Test 4 : Repeat non-0 status for 24 hours

Assume for this test that it starts immediately after the agent publishes a check result for CPU Usage at 85% and then remains at 85% for 24+ hours

The repeating check result status means that "Status" is equal to "LastPublishedStatus" so no repetition counter is evaluated.

Muting threshold is only evaluated when Status and LastPublishedStatus are equal to each other and not 0.

Instead, this test is only about Muting Threshold, so we will consider "LastTimeSent" instead of "RepeatCounter"

Note: The TimeStamp in use here is using the Unix Epoch format

The calculation for muting threshold begins by subtracting "LastTimeSent" from the current timestamp, and comparing the result to the value of MID Property "sn_agent.agent.time_to_resend_unchanged_non_info_status_sec" (or 86400 if not set).

To illustrate, I will describe these as follows for the below example:

"TimeToResend" = the value of MID Property "sn_agent.agent.time_to_resend_unchanged_non_info_status_sec" - in this example it is not set, so we will use 86400

"TimeSinceSent" = (Current TimeStamp) - (LastTimeSent)

These conditions must all be met to publish for Muting Threshold:

"LastPublishedStatus" == "Status" "Status" != 0 "TimeSinceSent" > "TimeToResend"

Check status: 1 (Warning)

MID Property for re-send time is not set, so defaults to 86400 seconds (1 day)

Event Check Status:

"LastStatus" = 1 "LastPublishedStatus" = 1 "Status" = 1 "LastTimeSent" = 1730308410 Current TimeStamp: 1730308530

Note: "Current TimeStamp" is not written in the logs, but is calculated at the time of evaluation - you may translate the log timestamp to Unix epoch to see the timestamp in this format.

Evaluation:

"TimeSinceSent" = 1730308530 - 1730308410 = 120

"TimeToResend" = 86400

"TimeSinceSent" (120)

Not enough time has passed since the last sent event, therefore the check result is ignored.

The agent log will show:

[INFO] [agent] check result is being ignored...

Check repeats with same status output for 24 hours - during this time, the check result is not published for any other reason.

On the first check result after 24 hours since last sent event, this is the evaluation:

Event Check Status:

"LastStatus" = 1 "LastPublishedStatus" = 1 "Status" = 1 "LastTimeSent" = 1730308410 Current Timestamp: 1730394840

Evaluation:

"TimeSinceSent" = 1730394840 - 1730308410 = 86430

"TimeToResend" = 86400

"TimeSinceSent" (86420) > "TimeToResend" (86400)

As enough time has passed since the last sent event, the check will be published if the Status conditions are correct too:

"LastStatus" == "LastPublishedStatus" == 1

Therefore, all the conditions have been met to publish based on Muting Threshold

The agent log will show:

[DEBUG] [agent] shouldPublishCheckResult: commandChanged = false, repetitionReached = false, mutingThresholdReached = true