Agent Client Collector Visibility - Support and Troubleshooting

Agent Client Collector Visibility For information about the Agent Client Collector, not specific to visibility/discovery, please see:

Agent Client Collector Agent Client Collector for Visibility (ACC-V) is a ServiceNow Agent installed on your Windows and Linux servers to run push-based Discovery on a host. ACC-V deploys Ruby scripts that execute OS Query commands and OS-specific commands to gather the information. You can discover data from various file systems and storage devices, TCP connections, running processes, and other information about target hosts.

You can register and manage your target systems in the ServiceNow Configuration Management Database (CMDB) using the ACC-V push-based model. There is no need to provide credentials, configure schedules, or scan IP ranges.

ACC-V is an extra mechanism to perform Discovery. It is an alternative to horizontal IP-based Discovery for OS-related attributes such as system information, network interfaces, running process. ACC-V is suitable for on-prem servers and cloud instances. ACC-V requires installation of ServiceNow Agent Client Collector (ACC) on the target host.

Starting in Quebec patch 3, the discovery_source, ACC-Visibility, was introduced to specifically denote that the CI was discovered by ACC-V.

For details on using push discovery and SAM together, please refer to:

Using push-based Discovery and SAM together Table of Contents Benefits Requirements How ACC-V works Troubleshooting Benefits The Agent initiates the communication and stays connected to the ServiceNow instance via the MID Server. The Agent(s) installed with ACC are self-aware, package the discovery data, and send this data back to the ServiceNow instance via the MID Server on a pre-determined schedule, as set by the ACC-V policy. If an interesting event occurs on the target host (such as coming back online after a prolonged shutdown), Discovery is triggered. The application is already tracking specific targets that have installed ACC and communicate with the MID Server. Requirements Discovery [com.snc.discovery] plugin must be installed and activated. Agent Client Collector Framework (ACC-F) must be installed on the ServiceNow instance. The Agent must be installed on the target host. Make sure you have upgraded to Quebec patch 3 or later. ITOM-Discovery or ITOM-Visibility SKUs (SU-based licensing) is required How ACC-V works At a high level, ACC-V works in coordination with the following:

ServiceNow Instance ServiceNow MID Server Target host machines ServiceNow Agent Client Collector which runs on said Target host machines ACC-V Architecture

ACC-V applies Checks and Policies to schedule and execute Discovery. Discovery is triggered during the following cases: Periodic scheduling: A policy-based approach where Discovery is triggered on a periodic basis On CI delete: When the computer or server CI record is deleted The "Enhanced Discovery – On CI Delete" business rule triggers the Endpoint Discovery Check with the CI associated with a given CI is deleted from sn_agent_cmdb_ci_agent. MID Server cycle: When the MID Server goes down and comes back up Target host cycle: When the target host goes down and comes back up Network break: When there is a break in the network link to the target How often discovery of the agents run / How is the discovery triggered As configured in the "Enhanced Discovery Policy" To view the policy navigate to "Agent Client Collector > Configuration > Policies" The check instance "Override interval" can override the policy interval No ecc queue output is created when discovery is triggered by the policy Discovery can be run on demand by opening the agent record and clicking on "Run Discovery" When triggered on demand an ecc_queue output record is created of topic "MonitoringProbe" and name "on_demand_request" The output payload will contain the required information necessary to run the discovery, including agent and policy to run In the payload you can see "clients_cis" in the json which is the agent ID of the sn_agent_cmdb_ci_agent How are the inputs processed Business Rule "AgentNowResponseProcessor" processes the ecc_queue input The Business Rule calls Script Include AgentNowHandler.processEccRecord(current); The script include uses the payload to get the check_type_id and query sn_agent_check_type For discovery it would be the types under sn_agent_disco_check_type The check type id is used to call the script to process the payload EnhancedDiscoveryHandler.handleDiscovery(checkResults) Some other script include used: MainDiscoveryHandler EnhancedDiscoveryHandler EndpointDiscoveryCloudHelper The scripts build a final JSON payload which is passed to the Identification and Reconciliation Engine Properties sn_agent.disco_minimum_threshold_for_rediscovery_minutes To avoid discovering the system too frequently sn_agent.disco_disable_ci_clobber_of_agentless_disco To avoid conflicts, ACC-V does not perform Discovery when the target host’s IP already has a CI that has been discovered by horizontal IP-based Discovery (discovery_source is ServiceNow). To override this behavior, set the system property sn_agent.disco_disable_ci_clobber_of_agentless_disco to false sn_agent.disco_ci_clobber_of_agentless_disco_threshold_days This setting is ignored if horizontal IP-based Discovery was not performed over a specific time. Update the system property [sn_agent.disco_ci_clobber_of_agentless_disco_threshold_days] if necessary. The default is 14 days Troubleshooting Data returned successfully in ecc_queue input payload, however CI is not updated accordingly Note: Following steps may require admin privileges. Test any scripts in a non production instance before running any scripts in production.

Review System Logs / node logs when the payload is processed and look for errors Error: Resolve errors, action will depend on what the error was No Error: Continue to next step Debug input payload processing by Get the sys_id of the ecc_queue input (To find the correct input search the ecc_queue for input payloads which contain the Agent Id) Opening Script Include AgentNowHandler On the script section click on "Open Script Debugger" (that is the icon with a bug) Set a breakpoint on processEccRecord() function Navigate to "System Definition > Scripts - Background" Set scope to "sn_agent" Run following script // In the following line, place the sys_id of the ecc_queue input var payloadSysID = ""; var eccRecord = new GlideRecord('ecc_queue'); eccRecord.get(payloadSysID); var ANH = new AgentNowHandler(); ANH.processEccRecord(eccRecord); Optionally, add debug statements to scripts where you suspect there might be issues with Missing serial numbers and TCP connections for Linux devices In order for the Agent to retrieve the OS serial numbers and TCP connections along with associated running processes, sudo access for “dmidecode” and “ss” is required on Linux systems. For example, this content could be added to /etc/sudoers or to an individual file in /etc/sudoers.d/:

Cmnd_Alias AGENT_ACC_V = /usr/sbin/dmidecode,/usr/sbin/ss servicenow ALL=(root) NOPASSWD:AGENT_ACC_V Payload returns with errors in the output section Example Error:

"output" : "C:/Program Files/ServiceNow/agent-client-collector/embedded/lib/ruby/gems/2.7.0/gems/json-2.3.0/lib/json/common.rb:156:in `parse': 783: unexpected token at '' (JSON::ParserError) Troubleshooting Example 01 - Send debugs to file on host:

Update endpoint_discovery.rb Find following code $logger = Logger.new(STDOUT) $logger.level = options.log_level.nil? ? "FATAL" : options.log_level Replace with following code $logger = Logger.new('pathAccountCanWriteTo') #Comment out log level or set to debug #$logger.level = options.log_level.nil? ? "FATAL" : options.log_level Reproduce issue Review log file created Troubleshooting Example 02 - Add more debug statements:

Update the enhanced discovery check to pass a debug parameter, for example: endpoint_discovery.rb --compact --select=basic_inventory,installed_software,file_systems,serial_numbers,network_adapters,tcp_connections,storage_devices,running_processes --log-level DEBUG Update the check-allow-list.json file in the Agent Client Collector to allow the above command to run Run a new test discovery Review new output and acc.log file Optionally add more debug statements to the endpoint_discovery.rb script, or other scripts called by it Example adding more debugs: # Creating a logger to log information to C:\temp\accDebug.log # in a path the acc service user can write to $logger = Logger.new('C:\temp\accDebug.log') # Log additional information $logger.info('variableOrStringToGetMoreInformation') Review new log file created Troubleshooting Example 03 - Run check directly on host:

The files for the Enhanced Discovery Check are downloaded to the agent's cache directory and thus checks can be run directly on the agent as well for troubleshooting.

On the next image, we see the results of running osquery command:

osqueryi --json --logger_min_status 1 "select hostname,hardware_vendor,hardware_model,hardware_version,physical_memory,cpu_brand from system_info"

The ruby scripts could be run directly via the command line as well for troubleshooting. However, note that this is not exactly the same as when the ACC runs them and may have different results or errors.

The endpoint_discovery.rb script receives an array for parameter "--select". These are the names of other scripts that are called by the endpoint_discovery.rb. Each of those scripts will discover data in the server and return a JSON file. All those results are added together and returned to the instance. The full command:

endpoint_discovery.rb --compact --select=basic_inventory,installed_software,file_systems,serial_numbers,network_adapters,tcp_connections,storage_devices,running_processes In the following example we test one of the scripts called by endpoint_discovery.rb by:

Find the osqueryi.exe path in the ACC cache directory and add to the user path, if not there may be an error for file not found Open the command prompt with the ACC service user Run endpoint_discovery.rb with only basic_inventory passed and send the output to a file endpoint_discovery.rb --log_level DEBUG --compact --select=basic_inventory >> basicInventoryOutput.txt Review file, example output: