For information about the Agent Client Collector, not specific to visibility/discovery, please see:
Agent Client Collector for Visibility (ACC-V) is a ServiceNow Agent installed on your Windows and Linux servers to run push-based Discovery on a host. ACC-V deploys Ruby scripts that execute OS Query commands and OS-specific commands to gather the information. You can discover data from various file systems and storage devices, TCP connections, running processes, and other information about target hosts.
You can register and manage your target systems in the ServiceNow Configuration Management Database (CMDB) using the ACC-V push-based model. There is no need to provide credentials, configure schedules, or scan IP ranges.
ACC-V is an extra mechanism to perform Discovery. It is an alternative to horizontal IP-based Discovery for OS-related attributes such as system information, network interfaces, running process. ACC-V is suitable for on-prem servers and cloud instances. ACC-V requires installation of ServiceNow Agent Client Collector (ACC) on the target host.
Starting in Quebec patch 3, the discovery_source, ACC-Visibility, was introduced to specifically denote that the CI was discovered by ACC-V.
For details on using push discovery and SAM together, please refer to:
- The Agent initiates the communication and stays connected to the ServiceNow instance via the MID Server.
- The Agent(s) installed with ACC are self-aware, package the discovery data, and send this data back to the ServiceNow instance via the MID Server on a pre-determined schedule, as set by the ACC-V policy.
- If an interesting event occurs on the target host (such as coming back online after a prolonged shutdown), Discovery is triggered. The application is already tracking specific targets that have installed ACC and communicate with the MID Server.
- Discovery [com.snc.discovery] plugin must be installed and activated.
- Agent Client Collector Framework (ACC-F) must be installed on the ServiceNow instance.
- The Agent must be installed on the target host.
- Make sure you have upgraded to Quebec patch 3 or later.
- ITOM-Discovery or ITOM-Visibility SKUs (SU-based licensing) is required
How ACC-V works
At a high level, ACC-V works in coordination with the following:
- ServiceNow Instance
- ServiceNow MID Server
- Target host machines
- ServiceNow Agent Client Collector which runs on said Target host machines
ACC-V applies Checks and Policies to schedule and execute Discovery. Discovery is triggered during the following cases:
- Periodic scheduling: A policy-based approach where Discovery is triggered on a periodic basis
- On CI delete: When the computer or server CI record is deleted
- The "Enhanced Discovery – On CI Delete" business rule triggers the Endpoint Discovery Check with the CI associated with a given CI is deleted from sn_agent_cmdb_ci_agent.
- MID Server cycle: When the MID Server goes down and comes back up
- Target host cycle: When the target host goes down and comes back up
- Network break: When there is a break in the network link to the target
How often discovery of the agents run / How is the discovery triggered
- As configured in the "Enhanced Discovery Policy"
- To view the policy navigate to "Agent Client Collector > Configuration > Policies"
- The check instance "Override interval" can override the policy interval
- No ecc queue output is created when discovery is triggered by the policy
- Discovery can be run on demand by opening the agent record and clicking on "Run Discovery"
- When triggered on demand an ecc_queue output record is created of topic "MonitoringProbe" and name "on_demand_request"
- The output payload will contain the required information necessary to run the discovery, including agent and policy to run
- In the payload you can see "clients_cis" in the json which is the agent ID of the sn_agent_cmdb_ci_agent
How are the inputs processed
- Business Rule "AgentNowResponseProcessor" processes the ecc_queue input
- The Business Rule calls Script Include AgentNowHandler.processEccRecord(current);
- The script include uses the payload to get the check_type_id and query sn_agent_check_type
- For discovery it would be the types under sn_agent_disco_check_type
- The check type id is used to call the script to process the payload
- Some other script include used:
- The scripts build a final JSON payload which is passed to the Identification and Reconciliation Engine
- To avoid discovering the system too frequently
- To avoid conflicts, ACC-V does not perform Discovery when the target host’s IP already has a CI that has been discovered by horizontal IP-based Discovery (discovery_source is ServiceNow). To override this behavior, set the system property sn_agent.disco_disable_ci_clobber_of_agentless_disco to false
- This setting is ignored if horizontal IP-based Discovery was not performed over a specific time. Update the system property [sn_agent.disco_ci_clobber_of_agentless_disco_threshold_days] if necessary. The default is 14 days
Data returned successfully in ecc_queue input payload, however CI is not updated accordingly
Note: Following steps may require admin privileges. Test any scripts in a non production instance before running any scripts in production.
- Review System Logs / node logs when the payload is processed and look for errors
- Error: Resolve errors, action will depend on what the error was
- No Error: Continue to next step
- Debug input payload processing by
- Get the sys_id of the ecc_queue input (To find the correct input search the ecc_queue for input payloads which contain the Agent Id)
- Opening Script Include AgentNowHandler
- On the script section click on "Open Script Debugger" (that is the icon with a bug)
- Set a breakpoint on processEccRecord() function
- Navigate to "System Definition > Scripts - Background"
- Set scope to "sn_agent"
- Run following script
// In the following line, place the sys_id of the ecc_queue input
var payloadSysID = "";
var eccRecord = new GlideRecord('ecc_queue');
var ANH = new AgentNowHandler(); ANH.processEccRecord(eccRecord);
- Optionally, add debug statements to scripts where you suspect there might be issues with
Missing serial numbers and TCP connections for Linux devices
In order for the Agent to retrieve the OS serial numbers and TCP connections along with associated running processes, sudo access for “dmidecode” and “ss” is required on Linux systems. For example, this content could be added to /etc/sudoers or to an individual file in /etc/sudoers.d/:
Cmnd_Alias AGENT_ACC_V = /usr/sbin/dmidecode,/usr/sbin/ss
servicenow ALL=(root) NOPASSWD:AGENT_ACC_V
Payload returns with errors in the output section
"output" : "C:/Program Files/ServiceNow/agent-client-collector/embedded/lib/ruby/gems/2.7.0/gems/json-2.3.0/lib/json/common.rb:156:in `parse': 783: unexpected token at '' (JSON::ParserError)
Troubleshooting Example 01 - Send debugs to file on host:
- Update endpoint_discovery.rb
- Find following code
$logger = Logger.new(STDOUT)
$logger.level = options.log_level.nil? ? "FATAL" : options.log_level
- Replace with following code
$logger = Logger.new('pathAccountCanWriteTo')
#Comment out log level or set to debug
#$logger.level = options.log_level.nil? ? "FATAL" : options.log_level
- Reproduce issue
- Review log file created
Troubleshooting Example 02 - Add more debug statements:
- Update the enhanced discovery check to pass a debug parameter, for example:
endpoint_discovery.rb --compact --select=basic_inventory,installed_software,file_systems,serial_numbers,network_adapters,tcp_connections,storage_devices,running_processes --log-level DEBUG
- Update the check-allow-list.json file in the Agent Client Collector to allow the above command to run
- Run a new test discovery
- Review new output and acc.log file
- Optionally add more debug statements to the endpoint_discovery.rb script, or other scripts called by it
- Review new log file created
Troubleshooting Example 03 - Run check directly on host:
The files for the Enhanced Discovery Check are downloaded to the agent's cache directory and thus checks can be run directly on the agent as well for troubleshooting.
On the next image, we see the results of running osquery command:
osqueryi --json --logger_min_status 1 "select hostname,hardware_vendor,hardware_model,hardware_version,physical_memory,cpu_brand from system_info"
The ruby scripts could be run directly via the command line as well for troubleshooting. However, note that this is not exactly the same as when the ACC runs them and may have different results or errors.
The endpoint_discovery.rb script receives an array for parameter "--select". These are the names of other scripts that are called by the endpoint_discovery.rb. Each of those scripts will discover data in the server and return a JSON file. All those results are added together and returned to the instance. The full command:
endpoint_discovery.rb --compact --select=basic_inventory,installed_software,file_systems,serial_numbers,network_adapters,tcp_connections,storage_devices,running_processes
In the following example we test one of the scripts called by endpoint_discovery.rb by:
- Find the osqueryi.exe path in the ACC cache directory and add to the user path, if not there may be an error for file not found
- Open the command prompt with the ACC service user
- Run endpoint_discovery.rb with only basic_inventory passed and send the output to a file
endpoint_discovery.rb --log_level DEBUG --compact --select=basic_inventory >> basicInventoryOutput.txt
- Review file, example output: