Agent Client Collector Visibility


For information about the Agent Client Collector, not specific to visibility/discovery, please see:

Agent Client Collector  for Visibility (ACC-V) is a  ServiceNow  Agent installed on your  Windows  and Linux servers to run push-based Discovery on a host. ACC-V deploys Ruby scripts that execute OS Query commands and OS-specific commands to gather the information. You can discover data from various file systems and storage devices, TCP connections, running processes, and other information about target hosts.

You can register and manage your target systems in the ServiceNow Configuration Management Database (CMDB) using the ACC-V push-based model. There is no need to provide credentials, configure schedules, or scan IP ranges.

ACC-V is an extra mechanism to perform Discovery. It is an alternative to horizontal IP-based  Discovery for OS-related attributes such as system information, network interfaces, running process. ACC-V is suitable for on-prem servers and cloud instances. ACC-V requires installation of  ServiceNow  Agent Client Collector (ACC) on the target host.

Starting in Quebec patch 3, the discovery_source, ACC-Visibility, was introduced to specifically denote that the CI was discovered by ACC-V.

For details on using push discovery and SAM together, please refer to:

Benefits

Requirements

How ACC-V works

At a high level, ACC-V works in coordination with the following:

ACC-V Architecture

This block diagram shows the components that work together

ACC-V applies Checks and Policies to schedule and execute Discovery. Discovery  is triggered during the following cases:

How often discovery of the agents run / How is the discovery triggered

How are the inputs processed

  1. Business Rule "AgentNowResponseProcessor" processes the ecc_queue input
  2. The Business Rule calls Script Include AgentNowHandler.processEccRecord(current);
  3. The script include uses the payload to get the check_type_id and query sn_agent_check_type
    • For discovery it would be the types under sn_agent_disco_check_type
  4. The check type id is used to call the script to process the payload
    • EnhancedDiscoveryHandler.handleDiscovery(checkResults)
    • Some other script include used:
      • MainDiscoveryHandler
      • EnhancedDiscoveryHandler
      • EndpointDiscoveryCloudHelper
  5. The scripts build a final JSON payload which is passed to the Identification and Reconciliation Engine

Properties

Troubleshooting

Data returned successfully in ecc_queue input payload, however CI is not updated accordingly

Note: Following steps may require admin privileges. Test any scripts in a non production instance before running any scripts in production.

  1. Review System Logs / node logs when the payload is processed and look for errors
    1. Error: Resolve errors, action will depend on what the error was
    2. No Error: Continue to next step
  2. Debug input payload processing by
    1. Get the sys_id of the ecc_queue input (To find the correct input search the ecc_queue for input payloads which contain the Agent Id)
    2. Opening Script Include AgentNowHandler
    3. On the script section click on "Open Script Debugger" (that is the icon with a bug)
    4. Set a breakpoint on processEccRecord() function
    5. Navigate to "System Definition > Scripts - Background"
    6. Set scope to "sn_agent"
    7. Run following script
      // In the following line, place the sys_id of the ecc_queue input
      var payloadSysID = "";
      var eccRecord = new GlideRecord('ecc_queue');
      eccRecord.get(payloadSysID);
      var ANH = new AgentNowHandler(); ANH.processEccRecord(eccRecord);
  3. Optionally, add debug statements to scripts where you suspect there might be issues with

Missing serial numbers and TCP connections for Linux devices

In order for the Agent to retrieve the OS serial numbers and TCP connections along with associated running processes, sudo access for “dmidecode” and “ss” is required on Linux systems. For example, this content could be added to /etc/sudoers or to an individual file in /etc/sudoers.d/:

Cmnd_Alias AGENT_ACC_V = /usr/sbin/dmidecode,/usr/sbin/ss 
servicenow ALL=(root) NOPASSWD:AGENT_ACC_V

Payload returns with errors in the output section

Example Error:

"output" : "C:/Program Files/ServiceNow/agent-client-collector/embedded/lib/ruby/gems/2.7.0/gems/json-2.3.0/lib/json/common.rb:156:in `parse': 783: unexpected token at '' (JSON::ParserError)

Troubleshooting Example 01 - Send debugs to file on host:

  1. Update endpoint_discovery.rb
  2. Find following code
    $logger = Logger.new(STDOUT)
    $logger.level = options.log_level.nil? ? "FATAL" : options.log_level
  3. Replace with following code
    $logger = Logger.new('pathAccountCanWriteTo')
    #Comment out log level or set to debug
    #$logger.level = options.log_level.nil? ? "FATAL" : options.log_level
  4. Reproduce issue
  5. Review log file created

Troubleshooting Example 02 - Add more debug statements:

  1. Update the enhanced discovery check to pass a debug parameter, for example:
    endpoint_discovery.rb --compact --select=basic_inventory,installed_software,file_systems,serial_numbers,network_adapters,tcp_connections,storage_devices,running_processes --log-level DEBUG
  2. Update the check-allow-list.json file in the Agent Client Collector to allow the above command to run
  3. Run a new test discovery
  4. Review new output and acc.log file
  5. Optionally add more debug statements to the endpoint_discovery.rb script, or other scripts called by it
    • Example adding more debugs:
      # Creating a logger to log information to C:\temp\accDebug.log 
      # in a path the acc service user can write to
      $logger = Logger.new('C:\temp\accDebug.log')
      # Log additional information
      $logger.info('variableOrStringToGetMoreInformation')
  6. Review new log file created

Troubleshooting Example 03 - Run check directly on host:

The files for the Enhanced Discovery Check are downloaded to the agent's cache directory and thus checks can be run directly on the agent as well for troubleshooting.

On the next image, we see the results of running osquery command:

osqueryi --json --logger_min_status 1 "select hostname,hardware_vendor,hardware_model,hardware_version,physical_memory,cpu_brand from system_info"

The ruby scripts could be run directly via the command line as well for troubleshooting. However, note that this is not exactly the same as when the ACC runs them and may have different results or errors.

The endpoint_discovery.rb script receives an array for parameter "--select". These are the names of other scripts that are called by the endpoint_discovery.rb. Each of those scripts will discover data in the server and return a JSON file. All those results are added together and returned to the instance. The full command:

endpoint_discovery.rb --compact --select=basic_inventory,installed_software,file_systems,serial_numbers,network_adapters,tcp_connections,storage_devices,running_processes

In the following example we test one of the scripts called by endpoint_discovery.rb by:

  1. Find the osqueryi.exe path in the ACC cache directory and add to the user path, if not there may be an error for file not found
  2. Open the command prompt with the ACC service user
  3. Run endpoint_discovery.rb with only basic_inventory passed and send the output to a file
    endpoint_discovery.rb --log_level DEBUG --compact --select=basic_inventory >> basicInventoryOutput.txt
  4. Review file, example output: