Agent Client Collector - Support and Troubleshooting

Agent Client Collector Summary Table of Contents The ACC Applications Benefits Features High-Level Architecture Setup and Deployment Extensibility Performance & Footprint Commands and Logs Configuration Agent Configuration Tables Instance Scheduled Jobs: Checks Policies Agent Client Collector Plugins / Assets Web Server MID Web Server Fail Over Configuration Lifecycles How are the agents (sn_agent_cmdb_ci_agent and sn_agent_ci_extended_info) records created and updated? How are the policies and checks updated in the agent? How are the policies and checks updated to the MID Web Server? How are assets downloaded to the agent? How are results sent from the agent to the instance? Troubleshooting Agent Down Instance Clones Q&A The Agent Client Collector (ACC) is a Sensu based agent which supports multiple use-cases. It can be used to monitor hosts where the agent is installed, monitor endpoints, discover the hosts where the agent is installed, and more.

The ACC Applications Agent-Client-Collector Framework (ACC-F) The ACC-F is a store app that manages the agent fleet and provides UI as well as API for SN apps to interact with the agent program Agent-Client-Collector Monitoring (ACC-M) The ACC-M is a store app running on top of the ACC-F and monitors OS and applications running on servers in order to feed events into the ServiceNow Event Management Agent-Client-Collector Visibility (ACC-V) The ACC-V is a store app running on top of ACC-F and collects information on servers and endpoints in order to populate the CMDB For more information on ACC-V please see Agent Client Collector Visibility Benefits Some of the benefits of the ACC are

End-to-end monitoring Ability to leverage advanced capabilities Efficient CI binding Reduced dependency on 3rd party monitoring tool One-stop shop for end-to-end AIOps solution Credentials for agent hosts do not need to be stored on the instance Features Monitor the health of servers, applications, network devices etc Monitor non-servers devices (e.g. network devices) can be done using proxy agent that will send SNMP/REST calls to the monitored device Entry points monitoring using remote HTTP calls from proxy agent Metrics collector and anomaly detection using Operational Intelligence High-Level Architecture

Note: Clotho and Events on the image above represent some of the components used with ACC-M

Setup and Deployment

Install Required Plugins Install MID Servers Navigate to MID Servers and click on "Setup Agent Client Collector Listener" This will install the "MID Web Servers" which the agents will be configured to connect to in later steps Install Agents on hosts which will be monitored/discovered Installation can vary slightly depending on the instance version. For installation steps see: Agent Client Collector Installation The User/Password (If using user/password instead of key) combination for the command should match the user configured for the "MID Web Server", this is not the same as the MID server user Navigate to "Agent Client Collector > Deployment > Agent Downloads" to download the Agent Client Collector installer Configure policies and checks once MID, MID Web Server, and agents are installed and connected Note: Configuring policies and checks could be done outside this order. However, without MID Servers and agents setup it would not be possible to test them.

Extensibility The agent framework is designed to allow the addition of flows that collect information from agents and save it in SN database The framework provides the means for distributing to agent scripts/executables which are not part of the initial installation, AKA ‘Assets’ Once a command is executed and a payload is created, developers can handle this payload by running scripts on the MID, the instance or both The framework provides an API for running a check on a given agent, regardless if the check is part of a policy Performance & Footprint See the Agent Client Collector Footprint docs page for the ACC version installed:

Agent Client Collector Footprint Commands and Logs Windows Linux MacOS Log Folder C:\ProgramData\ServiceNow\agent-client-collector\ /var/log/servicenow/agent-client-collector/ /var/log/servicenow/agent-client-collector/acc.log Installation Folder C:\Program Files\ServiceNow\agent-client-collector\ /etc/servicenow/agent-client-collector/ /opt/servicenow/agent-client-collector/

Stop Via services, service Agent Client Collector sudo systemctl stop acc sudo /bin/acc stop

Start Via services, service Agent Client Collector sudo systemctl start acc sudo /bin/acc start Or at startup by configuring launchctl with "/Library/LaunchDaemons/com.sn.acc.plist" Configuration File \acc.yml /acc.yml /Library/Application Support/servicenow/agent-client-collector/acc.yml Allow list Path found in acc.yml parameter allow-list Path found in acc.yml parameter allow-list /Library/Application Support/servicenow/agent-client-collector/check-allow-list.json Configuration The agent can be used to monitor and discovery the host it is installed on. The actions performed by the agents are done via "Checks". An event can be created if a check fails and triggers an alert. The checks are grouped into policies. A policy will determine on what population of agents (Linux, windows, application, etc) to run the checks on as well as the frequency.

Agent Configuration The main configuration files for the agent are the acc.yml and check-allow-list.yml. The check-allow-list.yml determines what commands the agent is allowed to run. Once a check is created click on related link "Generate allow-list content", next update the check-allow-list.yml file with the values generated so that the check is allowed to run on the agent.

Log parameters:

log-file The log file location. Default = Windows: C:/ProgramData/ServiceNow/Agent-client-collector/log Linux: /var/log/servicnow/agent-client-collector/acc.log

log-file-and-stdout Write logs to the stdout file. Default: false. log-file-max-age Maximum age, in days, of the log file before it is rotated out of the system memory. Default: 3. log-file-max-backups Maximum number of log files that can be stored before being rotated out of the system. Default: 3. log-file-max-size Maximum size, in MB, of the log file before it is rotated out of the system memory. Default: 10. log-level The log level to be measured by the logs. Available options are: Panic, Fatal, Error, Warn, Info, Debug. Default: Info. The specified log level represents the lowest level of events displayed in the log. For example, a user who specifies Error sees all Error events, as well as Fatal and Panic events.

Tables https://yourInstance.service-now.com/sys_db_object_list.do?sysparm_query=nameLIKEsn_agent

Instance Scheduled Jobs: https://yourInstance.service-now.com/sysauto_script_list.do?sysparm_query=sys_package%3Ddeb59787c317030039a3553a81d3aee9&sysparm_view=

Agent Client Collector keepalive Calculate Agent Feature Support Status Refresh And Publish Monitoring Policies Sync MID servers and Delete Drafts after policy import Update Mid List For All Agents Update Mid List For All New Agents Update Processed Checks of Request Checks A check is a combination of a command and its configuration. The check is executed on the Agent Client Collector's servers. Checks are provided with the base system, and their commands execute scripts which provide monitoring data for your operating systems and applications. A check's default name indicates what is being monitored and measured, the entity, and the monitoring data. For example, a check named os.linux.check-system-cpu checks the CPU data on a Linux system. The identified command in the check runs on the monitored server, providing an output and status.

Create a Check Definition Policies Policies consist of the CIs monitored by the Agent Client Collector and the checks that run on those CIs. When creating a policy, you configure a filter which determines the specific CIs on which the checks are to run. For example, you can configure a policy to run checks on all Apache web servers. You can create new policies or edit the default policies, as needed.

Create a New Policy Policy Lifecycle

Agent Client Collector Plugins / Assets An Agent Client Collector plugin is a script or group of scripts which provides additional agent capabilities. For example, collecting more metrics, performing more checks, or generating events when an application queue size is 60% or 80% full. You associate a check with a plugin by creating a dependency between the check and the plugin. A plugin can have a dependency with several checks at a time; similarly, checks can depend on several plugins at a time. Plugins run on the same agent as the check.

You create Agent Client Collector plugins, as needed. Plugins are formatted as tar.gz files and run together with their associated check.

The plugins/assets can be seen on table sn_agent_asset, or by navigating to "Agent Client Collector > Configuration > ACC Plugins".

Create and Edit Plugins Web Server The Web Server is the "endpoint" to where the agents will connect. To view the web servers navigate to "Agent Client Collector > Deployment > MID Web Servers". There you will see the list of web servers running. Those are the servers to which the agents can connect, as well as the port.

Via the related links in this form you can stop, start, restart, test and update parameters.

MID Web Server Fail Over Configuration The Agent's configuration file, acc.yml, determines to what MID Web Server it will connect. The configuration file for the agent allows for multiple Web Servers to be configured. The next MID Web Server configured in the acc.yml file will be used when communication cannot be established to a MID Web Server. The agents can also connect to a virtual IP address behind a load balancer.

Specify one or more MID server failover URLs in the acc.yml. The agent will communicate with the MID server using these URLs. This list is iterative; meaning the agent will try the first URL, if failed will move to the second URL and so on. If Auto-MID-Selection feature is on (the file mid.list.json is present), the agent will perform a connectivity test and will re-write the backend-url list. The order of the list will be based on ping time and then the number of agents already connected to the MID. This means that if the feature is enabled: the first MID server in the list should be the one with the lowest ping and the lowest number of agents.

To disable sending periodic MID Server updates from the ServiceNow instance to existing agents: Navigate to System Properties > All Properties. Set the sn_agent.enable_auto_mid_selection property to false.

To disable automatic MID Server selection for individual agents: In the agent's acc.yml file, locate the enable_auto_mid_selection property. Set the property value to false. Lifecycles How are the agents (sn_agent_cmdb_ci_agent and sn_agent_ci_extended_info) records created and updated? Note: On the agent set acc.yml file set log-level = "debug", and set mid.log.level = debug on the MID server, to see detailed debug messages.

The agent uses the configuration from the acc.yml to connect to the MID Web Server The agent sends keepalive events to ip:port/ws/events, when the agent first starts up you can see the keepalive loop starting: {"component":"agent","level":"info","msg":"Starting keepalive loop","time":"2021-07-26T09:43:49-07:00"} On the MID Server agent logs we can see the MID Server receiving the request and the reply: 07/26/21 10:29:53 (772) qtp540917385-156 DEBUG: (156) com.service_now.mid.webserver.jetty.WebServer - SERVER onFillable() ... 07/26/21 10:29:53 (773) qtp540917385-156 DEBUG: (156) com.service_now.mid.webserver.jetty.WebServer - onBinaryMessage(HeapByteBuffer@4079c03a[p=0,l=2293,c=2293,r=2293]={ >>}) 07/26/21 10:29:53 (791) qtp540917385-156 DEBUG: handleKeepalive: payload [{"timestamp":1627320593,"entity":{"entity_class":"agent","system":{"hostname":"......"}}}] The MID Server sends an update to the instance (Update "Last refreshed" field or any other fields which need to be updated) On the instance:

The Agent Client Collector API (/api/sn_agent/agents) receives the update sent by the MID server and updates the agent information

The API is defined in /sys_ws_definition.do?sys_id=cf0d4208c3e3030039a3553a81d3ae9a

The REST resource will update/create the agent accordingly

Finally the MID Server replies to the agent: 07/26/21 10:29:53 (791) qtp540917385-156 DEBUG: Publishing message [{ }] to client:WIN-IHKAN2LQJE1. remote: org.eclipse.jetty.websocket.jsr356.JsrAsyncRemote@3efeb957 07/26/21 10:29:53 (792) qtp540917385-156 DEBUG: (156) com.service_now.mid.webserver.jetty.WebServer - sendText("keep_alive_response { }") On the agent we can see: {"component":"agent","content_type":"application/json","level":"debug","msg":"message received","payload_size":3,"time":"2021-07-26T09:46:54-07:00","type":"keep_alive_response"} {"component":"agent","level":"debug","msg":"keepalive response received from the backend. Setting the read deadline to 1627318314","time":"2021-07-26T09:46:54-07:00"} Note: Scheduled job "Agent Client Collector keepalive" runs every minute to check for agents which the last_refreshed field has not been updated since the last keepalive and sets its statuses accordingly.

How are the policies and checks updated in the agent? As part of the keep alive process, the MID Server checks for updates to the agent configuration (updates to checks, policies, etc) The MID Server will reply to the keepalive and send additional information if there are updates to be sent to the agent. In the MID Server agent logs we can see, for example: 07/26/21 12:32:53 (776) qtp540917385-157 DEBUG: Publishing config {"checksum":"826562631", "check_requests":[{ ... }]} to client: name { ... }, agent_id 5d06b30ec6c4ce0b How are the policies and checks updated to the MID Web Server? Job "Refresh And Publish Monitoring Policies" (sysauto_script.do?sys_id=126cee2bc3b513002a6f741e81d3ae87) runs every minute The job calls MonitoringConfig.syncPoliciesToMid(); which checks if we need to send an update to the MID Servers If an update must be sent to the MID server, create an ecc_queue output record with topic "MonitoringProbe" and source = "config_publish" The MID server process the output and updates the agent policies How are assets downloaded to the agent? The assets files are synchronized to the MID servers via the FileSyncer The FileSyncer is a process/thread that keeps MID Server files synchronized to instance files The assets can be seen in the MID server folder: %INSTALL_FOLDER%/agent/static/acc_plugin The keepalive process which keeps the agent record on the instance up to date, as well as policy/checks on the agent client, also contain an "assets" section The assets section of the payload tells the agent what assets need to be downloaded, for example: "assets" : [ { "sha512" : "c5149676070a227ab75a6c2979a568404e6710c9c8d57766d85a7c759504b833788ad133b99caa8d53cbd134fe42817b84eef1880b41d29a8e0e2f51fe8d5c73", "url" : "{{MID_URL}}/static/acc_plugin/windows/all/all/all/acc-visibility-modules-windows.tar.gz", "metadata" : { "name" : "acc-visibility-modules-windows", "namespace" : "default" } } The agent downloads the assets from IP:PORT/static In the agent, the files are downloaded to its "cache" folder Windows: C:/ProgramData/ServiceNow/Agent-client-collector/cache Linux: /var/log/servicnow/agent-client-collector/cache This file synching uses a pre-existing MID Server sync feature. Detailed information on how this works in general, known issues, and debugging tips can be found in: KB0852276 How MID Server File Synchronisation works, to help when Troubleshooting

How are results sent from the agent to the instance? The checks will run as configured in the policies The agent sends the check result to the MID Server using the WebSocket connection On the agent acc.log we can see the result being run and sent, for example: {"component":"agent","level":"info","msg":"Running check, name: policy: Windows OS Metrics, check:os.windows.metrics-system-cpu-load ... {"component":"agent","level":"debug","msg":"{winchecks metric-windows-cpu-load [AGENT_DIR=C:\\Program Files\\ServiceNow\\agent-client... {"component":"agent","content_type":"application/json","level":"info","msg":"sending message","payload":"{\"timestamp\":1628002490,\"... On the MID Server we can see the result received DEBUG: handle result from client [WIN-IHKAN2LQJE1] agent_id [5d06b30ec6c4ce0b], check result is [{"timestamp":1628004950,"check":{"co... DEBUG: MID Script Include cache hit for name=MonitorResultParser On the MID Server the MID script include MonitorResultParser checks if the check result has the mid_script field populated, this will be configured on table sn_agent_check_type Is the mid_script field populated? Yes: Run the mid_script to handle the result Depending on the mid_script the result may, for example: Sent back to the ecc_queue as an input Sent back as an event Sent back as a metric No: Send the result to the ecc_queue Once the result is in the instance, the sn_agent_check_type instance script will process the result Troubleshooting Note: On the agent set acc.yml file set log-level = "debug", and set mid.log.level = debug on the MID server, to see detailed debug messages.

Agent Down Make note of the MID web server the agent should connect to, this can be seen on the agent list or via table sn_agent_ci_extended_info Is the MID server up? Yes: Continue to next steps No: Start the MID server Did MID Server start successfully and show as up? Yes: Continue to next steps No: Troubleshoot MID server startup issues. See: MID Server Down Troubleshooting Navigate to "Agent Client Collector > Deployment > MID Web Server" Is the MID web server started? Yes: Continue to next steps No: Click on "Start" to start the MID web server Did MID web server start successfully and the operating system shows a process listening on the configured port? Yes: Continue to next steps No: Review the MID server agent and wrapper logs for issues starting the web server Is the agent running on the host? (Can be checked via services.msc on windows or via tools like top or ps on unix/linux operating systems) Yes: Review the acc.log file No: Review the acc.log file Does the acc.log display any network errors? Yes: Check that the server where the agent is running can communicate too the web server host and port configured Resolve any other errors (the action to be taken will depend on the error) Instance Clones MID Server and ACC installs are not Clones when an instance is Cloned over another instance, and their directly related files and configurations are preserved (or should be, but might not be), but a lot of code and settings is cloned over from the source instance. More details on the known issues, and how Clones work with ACC can be found in:

KB1002549 Agent Client Collector and Clones KB0786475 MID Servers and Clones Q&A What happens when the agent cannot connect? The agent will queue up information until it can connect to the MID server again. What happens if agent sends data to MID, however MID server cannot connect to the instance? For checks which write to the ecc_queue, the MID server uses the same mechanism to send data to the instance and will retry up to so many times. What is the Agent Client Collector for windows user account password? When installing the ACC for windows you can chose an existing account or let the installation create an account. The installation creates local account called "servicenow". This account password is dynamically generated. The password is not needed as the service can be started and stopped via windows services. If the password must be updated, this can be done via the windows user configuration, and then the service can be stopped and updated in order to have matching passwords. [back to top]