Amazon SSM Discovery - Troubleshooting Guide - Support and Troubleshooting

Amazon SSM Discovery - Troubleshooting Guide Summary .ns-kb-css-body-editor-container { p { font-size: 12pt; font-family: Lato; color: #000000; } span { font-size: 12pt; font-family: Lato; color: #000000; } h2 { font-size: 24pt; font-family: Lato; color: black; } h3 { font-size: 18pt; font-family: Lato; color: black; } h4 { font-size: 14pt; font-family: Lato; color: black; } a { font-size: 12pt; font-family: Lato; color: #00718F; } a:hover { font-size: 12pt; color: #024F69; } a:target { font-size: 12pt; color: #032D42; } a:visited { font-size: 12pt; color: #00718f; } ul { font-size: 12pt; font-family: Lato; } li { font-size: 12pt; font-family: Lato; } img { display: block; max-width: ; width: auto; height: auto; } } Overview

This guide provides instructions on how to effectively troubleshoot the integration of AWS Systems Manager (SSM) within the ServiceNow platform. The focus is on enabling technical support teams to efficiently discover virtual machines (VMs) and their associated configuration items (CIs) within the AWS cloud environment. It is designed to help users identify potential issues, verify correct operations, and debug complex discovery workflows that leverage AWS SSM agents to gather critical information from managed instances.

Cloud Discovery Architecture

Within the AWS discovery framework used by ServiceNow, there are two primary schedules that govern the process of identifying and gathering information about cloud resources:

Cloud Resource Schedule : This schedule is responsible for the initial identification and enumeration of cloud resources, such as EC2 instances, security groups, and other AWS components. It focuses on high-level discovery to provide an overview of available cloud assets. VM Schedule : This second schedule takes a deeper dive into the detailed discovery of virtual machines themselves. It collects granular data including operating system details, installed software, network configurations, and other VM-specific attributes.

Notably, AWS Systems Manager (SSM) integration is utilized exclusively within the VM Schedule to perform detailed discovery tasks

Prerequisites

AWS Systems Manager (SSM) integration is supported on ServiceNow Zurich release and later versions. To ensure full functionality, make sure these three apps are updated to at least the following versions on your Zurich (or newer) instance:

CMDB CI Class Models (app-cmdb-content) — version 1.74.0 Discovery and SM Patterns (app-itom-patterns) — version 1.27.0 Cloud Discovery Workspace (app-itom-cloud-workspace) — version 1.7.1

Discovery Schedule Property

Make sure that the discovery agent property within the VM schedule is explicitly set to AWS SSM. This setting directs the discovery process to leverage AWS Systems Manager for querying VMs. If this property is instead set to MID, the system defaults to traditional MID server-based discovery methods, which do not utilize SSM.

Other Configurations

For the SSM-based discovery functionality to operate correctly, specific system properties and MID properties needs to be configured. Please refer to the SSM Setup documentation for the complete list.

Amazon AWS - SSM Cloud Agents (LP) pattern

A new discovery pattern named Amazon AWS - SSM Cloud Agents (LP) has been introduced. This pattern:

Initiates a REST API call to AWS to retrieve information on active SSM agents across the environment.

Populates a dedicated table within ServiceNow called Cloud System Management Agents, which maintains up-to-date records of discovered SSM agents.

If the SSM VM schedule is not discovering a particular VM, check whether the SSM agent associated with that VM’s IP address is present in the above table. SSM-based discovery can only occur on VMs that have the SSM agent installed and running .

Please note that hybrid SSM agents, i.e., agents running on non-EC2 instances, are not stored in the Cloud System Management Agents table, as this use case is not yet supported.

AWS Console Tools

Fleet Manager

AWS provides several native tools to assist in managing and verifying SSM agent status:

The Fleet Manager console (found under AWS Systems Manager) allows administrators to view all registered managed instances within a given AWS region.

By accessing Fleet Manager, support teams can verify which SSM agents are currently running and operational.

This list should correspond directly with the entries present in ServiceNow's Cloud System Management Agents table (except hybrid agents)

Run Command & Documents

What is a Document?

Documents are predefined sets of scripts that can be executed remotely on managed instances:

For example, the ServiceNow-RunShellScript document enables execution of shell commands on Linux-based VMs. Parameters can be passed as input to a document. For example, the above document includes parameters such as: command: The actual command to execute executionUser: Specifies the user account using which commands will be run (default is root if not specified) executionTimeout: Defines the maximum duration allowed for command execution before timing out. All ServiceNow provided SSM documents must be pre-deployed across all AWS regions where managed instances exist. ServiceNow currently supports only version 1.0 of each document. Any modifications made to the document will be ignored as the MID server always calls version 1.0 of the document. Customers must avoid modifying the documents. When a Linux document runs, if no execution user is specified, SSM runs the document with root privileges. Windows documents do not have an ‘execution user’ parameter. It always runs as a SYSTEM user.

Running a Document:

To execute a document manually for troubleshooting:

Navigate to AWS Console > Systems Manager > Run Command. Select the 1.0 document version Input any required parameters Choose the target instances on which to run the command. Execute the command and monitor the results. Each execution generates a unique Command ID that can be referenced for audit and debugging purposes.

See https://docs.aws.amazon.com/systems-manager/latest/userguide/run-command.html for more details

Troubleshooting Steps

Each step of a discovery pattern that uses SSM results in the invocation of a separate Run Command in AWS Systems Manager:

Each of these commands generates a unique Command ID As shown below, the pattern logs include the corresponding AWS Command ID for reference.

To review command histories and outputs, navigate to AWS Console > Systems Manager >Run Command > Command History and search by Command ID. If a particular step is failing, use the Run Command feature in the AWS Console or CLI (see below) to execute the command or script using the appropriate document. This helps determine whether the issue originates at the AWS level or the MID Server level . You can identify the document used to run the command by checking the Command History section—search by Command ID to locate the relevant details. The MID server processes result from AWS, so determining if the error happens on the AWS side or MID side will help with troubleshooting. If the root user is not used for discovery (applicable only for Linux documents) and the customer has specified a different user (see the credentials section in the setup documentation to learn how to setup non-root user ), make sure to run the document on the console/CLI using the same user. If the user does not have the necessary privileges to run a particular command, the execution will fail.

The ServiceNow-provided documents perform pre-processing on the input command before execution. To check whether this pre-processing is causing any issues, use Session Manager to connect directly to the machine, as shown below.

Run the same command manually on the terminal, after connecting to the machine, to check if it works without the document’s pre-processing logic.

Common Errors

When executing a high volume of concurrent SSM or S3 REST API calls, AWS may return client-side rate limiting errors such as:

a) Error sending command via AWSSSMAsyncClient: software.amazon.awssdk.services.ssm.model.SsmException: Rate exceeded

b) Error sending command via AWSSSMAsyncClient: software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request:

To improve reliability under load, a retry mechanism has been implemented for failed API calls. This mechanism is configurable through the following MID Server properties:

mid.discovery.aws_ssm.max_ssm_client_retries – Applies to SSM transactions. mid.discovery.aws_ssm.max_s3_client_retries – Applies to S3 REST calls, which are used to retrieve the results of Run Command invocations. The default number of retries is 3 , with a maximum of 10 . Values above 10 will be ignored.

In certain scenarios—such as an issue on the AWS backend causing AWS threads to hang—MID Server threads can also become stuck. In extreme cases, this could result in all MID threads hanging, rendering the MID Server completely unresponsive. To prevent threads from hanging indefinitely, a maximum timeout has been implemented for AWS API calls. If a call takes longer than the specified timeout, it will be terminated.

This timeout can be configured using MID Server properties:

mid.discovery.aws_ssm.ssm_api_call_timeout for SSM API calls mid.discovery.aws_ssm.s3_api_call_timeout for S3 REST API calls These properties accept an integer value representing the maximum timeout (in seconds). Internally, this value is passed to the apiCallTimeout() method defined in the AWS SDK’s ClientOverrideConfiguration . If a timeout exception is thrown, the REST API call will be retried. The number of retry attempts is controlled by the above-mentioned MID Server properties:

mid.discovery.aws_ssm.max_ssm_client_retries for SSM API calls mid.discovery.aws_ssm.max_s3_client_retries for S3 API calls

It was observed that, at times, the SSM agent on the EC2 instance may go down, resulting in REST API call failures. Please ensure that the agent is running by checking its status in Fleet Manager

Pattern Debugger

A toggle within the Pattern Debugger tool allows users to test discovery patterns directly through SSM:

This feature is accessed by selecting an IP address in the debugger interface and enabling the "Use AWS SSM" option. It only functions if the selected IP corresponds to an operational SSM agent in the Cloud System Management Agent table.

CLI

For quick verification from the command line, one can run AWS CLI commands such as:

aws ssm describe-instance-information \

--region us-east-2 \

--query "InstanceInformationList[*].[InstanceId, PlatformName, PlatformType]" \

--output table \

--no-cli-pager

This AWS CLI command lists all managed instances currently recognized by SSM and should align with the agents recorded in the Cloud System Management Agents table within ServiceNow.

aws ssm send-command \

--instance-ids "i-0123456789abcdef0" \

--document-name "AWS-RunShellScript" \

--parameters commands=["echo Hello World"] \

--region us-east-2

Here’s exactly what it does:

Uses the AWS-RunShellScript document — which runs shell commands on Linux-based instances. Runs the command: echo Hello World on the EC2 instance with ID i-0123456789abcdef0 in the us-east-2 region. Sends this command to the instance for immediate execution. The SSM Agent on the instance executes the shell command and reports back the result.

Release .ns-kb-css-body-editor-container { p { font-size: 12pt; font-family: Lato; color: #000000; } span { font-size: 12pt; font-family: Lato; color: #000000; } h2 { font-size: 24pt; font-family: Lato; color: black; } h3 { font-size: 18pt; font-family: Lato; color: black; } h4 { font-size: 14pt; font-family: Lato; color: black; } a { font-size: 12pt; font-family: Lato; color: #00718F; } a:hover { font-size: 12pt; color: #024F69; } a:target { font-size: 12pt; color: #032D42; } a:visited { font-size: 12pt; color: #00718f; } ul { font-size: 12pt; font-family: Lato; } li { font-size: 12pt; font-family: Lato; } img { display: block; max-width: ; width: auto; height: auto; } } This feature is available starting in Zurich release and also requires the following additional applications:

Mainline Applications Discovery ServiceNow Store Applications (May 2025 release or later) Cloud Discovery Workspace CMDB CI Class Models Discovery and Service Mapping Patterns