Understand Agent Client Collector record fields and troubleshoot empty CI references - Support and Troubleshooting

Understand Agent Client Collector record fields and troubleshoot empty CI references Summary .ns-kb-css-body-editor-container { p { font-size: 12pt; font-family: Lato; color: #000000; } span { font-size: 12pt; font-family: Lato; color: #000000; } h2 { font-size: 24pt; font-family: Lato; color: black; } h3 { font-size: 18pt; font-family: Lato; color: black; } h4 { font-size: 14pt; font-family: Lato; color: black; } a { font-size: 12pt; font-family: Lato; color: #00718F; } a:hover { font-size: 12pt; color: #024F69; } a:target { font-size: 12pt; color: #032D42; } a:visited { font-size: 12pt; color: #00718f; } ul { font-size: 12pt; font-family: Lato; } li { font-size: 12pt; font-family: Lato; } img { display: ; max-width: ; width: ; height: ; } } Agent Client Collector (ACC) records have several status fields and references to the configuration item (CI) and MID Server. Understanding how these fields are populated and updated helps you resolve issues with your ACCs.

Related tables The following tables store ACC data:

Agent Client Collectors [sn_agent_cmdb_ci_agent] : The main ACC record. This is a CMDB CI class. When referencing an ACC from another record, use this table. Agent Client Collector Info [sn_agent_ci_extended_info] : Stores additional fields. Each ACC record has a corresponding record in this table. Most status columns visible in the Agents list are dot-walked from this table. MID Server [ecc_agent] : The MID Server record that the ACC currently communicates through. Configuration item : The host server CI where the ACC is installed. Agent Client Collectors list columns The following columns appear in the default view for the sn_agent_cmdb_ci_agent list. The dot-walk path indicates which table each field comes from:

name agent_extended_info. status agent_extended_info. cmdb_ci (reference to cmdb_ci) agent_extended_info.cmdb_ci. sys_class_name agent_extended_info. mid (reference to ecc_agent) ip_address agent_extended_info. up_since agent_extended_info. running_checks_num agent_extended_info. data_collection agent_extended_info. is_duplicate agent_extended_info. last_refreshed agent_extended_info. agent_version Record creation and status updates When you install an ACC, the connected MID Server sends a REST request to the instance. This creates the sn_agent_cmdb_ci_agent and sn_agent_ci_extended_info record pair. After initial creation, the same REST message is sent every minute to keep records updated.

In the syslog_transaction table, this appears as a REST transaction created by the MID Server user with the URL /api/sn_agent/agents/updateClientTimestamp?api=api.

The Agent Client Collector API ( /api/sn_agent/agents ) is defined by the following records:

Scripted REST Web Service: /sys_ws_definition.do?sys_id=cf0d4208c3e3030039a3553a81d3ae9a updateClientTimestamp resource: /sys_ws_operation.do?sys_id=b64dc208c3e3030039a3553a81d3ae76 Script Include (MonitoringConfig): /sys_script_include.do?sys_id=6c62a3f0c305130039a3553a81d3ae88 The MonitoringConfig script include uses the updateClients and classifyClientsAndUpdate functions.

This API handles three situations:

Newly installed ACCs Full update Update timestamp and status only If the ACC is new, the insertNewClients function inserts the record pair. Otherwise, the updateClientKeepAlive function updates the values.

Fields updated in sn_agent_cmdb_ci_agent:

name agent_id ip_address Fields updated in sn_agent_ci_extended_info:

status mid up_since name agent_id ip_address is_windows agent_version Note : The Delete agent info business rule prevents orphan records by deleting the corresponding sn_agent_ci_extended_info record when you delete a sn_agent_cmdb_ci_agent record. If this rule is inactive, orphan sn_agent_ci_extended_info records remain and can cause empty sn_agent_cmdb_ci_agent records to be created. For example, if you delete an agent from the table but the agent is still installed, the next keep-alive finds the orphan sn_agent_ci_extended_info record and creates an empty sn_agent_cmdb_ci_agent record.

Host CI identification and reference After record creation, the ACC must discover itself to either create a new host server CI in the CMDB or link to an existing CI record.

The check-discovery-basic check performs this discovery using the check_discover.rb command: /sn_agent_check_def.do?sys_id=f9fe1e3c536113006dfeddeeff7b12a2

In the Paris release (ACC 2.2), this check definition requires the osquery plugin asset.

The hourly scheduled job Refresh host data for Agents runs for all agents that have an empty Host reference or have not been rediscovered within the time set in the system property sn_agent.host_data_collection_refresh_duration_seconds (default: 43200 seconds / 12 hours): /sysauto_script.do?sys_id=e59cc3d2531030100112ddeeff7b12f0

If you do not want to wait for the scheduled job, select Collect Host Data on the agent form.

A probe output is placed in the ECC Queue for the MID Server to run, which causes the ACC to discover itself. For initial discovery, depending on your setup, there might not be an output payload—only inputs.

The probe uses the following parameters:

agent: MID Server of the ACC topic: MonitoringProbe name: on_demand_request source: on_demand_request payload: Contains the check definition including name, command, interval, timeout, assets, and other parameters The input payload contains a JSON object with the ACC host's full details, including hostname, serial number, and other identifying information.

The sensor for this input is the AgentNowResponseProcessor business rule, which uses the processEccRecord function from the AgentNowHandler script include.

Because this is a Discovery type check ( /sn_agent_check_type.do?sys_id=4048774567633300b7b72dbd2685efa1 ), the DiscoveryHandler script include uses the handleDiscovery and createHostCis functions to create or update the host CI using the standard IRE APIs, then finds the ACC record and updates the cmdb_ci field.

Note : Beginning with the Quebec release (ACC-F 2.3.0), endpoint_discovery.rb is used instead of check_discover.rb by both ACC-F and ACC-V. Additional assets are required: acc-f-commons and acc-f-modules. This may change in subsequent releases.

Troubleshoot plugin download failures For any check that depends on plugin assets, the ACC must first download the plugin. In Paris (ACC 2.2), the osquery plugin ( /sn_agent_asset_name.do?sys_id=175dddb4533a1010b9ffddeeff7b1203 ) has OS-specific tarballs attached in its related records. The ACC requests the plugin from a URL such as:

https:// : /static/acc_plugin/linux/all/all/all/osquery.tar.gz

If the file cannot be downloaded (for example, due to security policies), the plugin is not synced to the agent, the check does not run, the probe fails, and the Host CI is not linked to the Agent record. This prevents most ACC functionality because monitoring events and metrics cannot be tied to the CI in the CMDB.

The acc.log file with debug enabled in the acc.yml configuration file may show errors such as:

{"component":"agent","error":"error getting plugin for check: sha512 'f60522ecfc7db6d18f50b6e317905e8e4bb737757c0b4b7642a53af19ca7aef3fcb349af36c643a216ab240d5f069797800f8ea6ba074207e3ff7854acbc1f5f' does not match specified sha512: 'd7f242d5bccefbac4c5cc0068aaef416a96424d6503393b5e45ca706f8fc1a3b924324f28f32140b95ad1ffbce0f8722dd0e2c54be38d5ab3c6675b339099e95'","level":"error","msg":"sending failure","time":"2021-04-07T04:01:54+01:00"}

This error is misleading. The SHA512 check compares the file contents with the SHA256 checksum for the plugin asset saved in the asset.db file. In this case, it compared the expected checksum with an HTML error page from a network device that blocked the request—not the actual file.

{"level":"info","msg":"finished fetching asset","status":"503 Service Unavailable","time":"2021-04-07T03:59:55+01:00"}

This message is also misleading because the MID Server was functioning correctly. However, both messages indicate that the ACC could not retrieve the osquery.tar.gz file, did not have the plugin asset, and could not run discovery.

To troubleshoot, use cURL or wget to replicate the request from the ACC host:

[root@acchost ~]# curl -Lk -u web_service_username:password https://10.x.x.x:8081/static/acc_plugin/linux/all/all/all/osquery.tar.gz File Transfer Blocked ... File Transfer Blocked Transfer of the file you were trying to download or upload has been blocked in accordance with company policy. Please contact your system administrator if you believe this is in error.

File name: osquery.tar.gz

...

If you receive an HTML response indicating "File Transfer Blocked," create exceptions in the software policy or firewall/proxy that is blocking the file type.

The file downloads to the temp folder of the ACC host (or the temp folder of the user running the ACC service). Investigate any restrictions or permissions issues for these file types in that folder. The plugin asset is staged in temp while security checks run, then extracted to the appropriate cache folder for the agent, and the temp files are deleted.

Troubleshoot check output errors A known error causes an unexpected result value in the ECC Queue input for the endpoint_discovery.rb check. The output contains a JSON::ParserError instead of the expected IRE payload.

Upgrade to ACC-F 2.5.0 or later to resolve this issue.

Other script errors may indicate different problems.

"output" : "C:/Program Files/ServiceNow/agent-client-collector/embedded/lib/ruby/gems/2.7.0/gems/json-2.3.0/lib/json/common.rb:156:in `parse': 783: unexpected token at '' (JSON::ParserError)\r\n\tfrom C:/Program Files/ServiceNow/agent-client-collector/embedded/lib/ruby/gems/2.7.0/gems/json-2.3.0/lib/json/common.rb:156:in `parse'\r\n\tfrom C:/ProgramData/ServiceNow/agent-client-collector/cache/acc-f-commons/bin/endpoint_discovery.rb:179:in `block in &lt;main&gt;'\r\n\tfrom C:/ProgramData/ServiceNow/agent-client-collector/cache/acc-f-commons/bin/endpoint_discovery.rb:166:in `each'\r\n\tfrom C:/ProgramData/ServiceNow/agent-client-collector/cache/acc-f-commons/bin/endpoint_discovery.rb:166:in `&lt;main&gt;'\r\n",

Troubleshoot IRE insert failures Example 1: Missing mandatory field This example uses ACC-F 2.5.0 on Quebec Patch 5, with the MID Server and ACC on a Windows Server 2019 instance.

Note that Data Collection is off. Selecting Resume data collection on the agent form does not resolve this issue.

Find the ecc_queue input for MonitoringProbe/on_demand_request where the input payload contains "name" : "check-discovery-basic" . The ecc_queue payload field should include a large JSON output with all the details of the ACC host server, including basic_inventory with platform, name, serial_number, and other identifying attributes.

"output" : "{\"basic_inventory\":{\"platform\":\"windows\",\"name\":\"cs-win8695\",\"manufacturer\":\"VMware, Inc.\",\"model_id\":\"VMware Virtual Platform\",\"model_number\":\"\",\"serial_number\":\"VMware-42 34 fd 2e 71 c2 80 c9-57 3c 4e e8 5e e9 d6 3f\",\"operating_system_domain\":\"supportlabwin.local\",\"operating_system\":\"Microsoft Windows Server 2019 Standard\",\"operating_system_version\":\"10.0.17763\",\"operating_system_service_pack\":\"\",\"operating_system_architecture\":\"64-bit\",\"os_address_width\":\"64\",\"ram\":\"4096\",\"cpu_manufacturer\":\"GenuineIntel\",\"cpu_name\":\"Intel(R) Xeon(R) Gold 5120 CPU @ 2.20GHz\",\"cpu_speed\":\"2195\",\"cpu_count\":\"1\",\"cpu_core_count\":\"1\",\"cpu_core_thread\":\"1\",\"kernel_release\":\"\",\"ip_address\":\"10.190.132.12\",\"default_gateway\":\"10.190.132.1\",\"is_virtual\":\"true\",\"fully_qualified_domain_name\":\"cs-win8695.supportlabwin.local\",\"object_id\":\"\",\"start_date\":\"1626852162000\",\"short_description\":\"\"},\"tcp_connections\":{\"connections\":[{\"type\":\"on\",\"pid\":\"860\",\"ip\":\"0.0.0.0\",\"port\":\"135\"},{\"type\":\"on\",\"pid\":\"4\",\"ip\":\"0.0.0.0\",\"port\":\"445\"},{\"type\":\"on\",\"pid\":\"404\",\"ip\":\"0.0.0.0\",\"port\":\"3389\"},{\"....

If the host attributes are not in the output, something went wrong when running the check in the ACC. Review the ACC logs.

If the data is present as in this example, the problem is likely with the CMDB Identification and Reconciliation Engine (IRE) inserting the CIs.

To enable full IRE debug logging, temporarily add the system property:

glide.cmdb.logger.source.identification_engine = info,warn,error,debug,debugVerbose

Select Run again on the ecc_queue input form to force the sensor to rerun.

Note : The Run again (debug) option, which enables Session Debug, does not work for ACC inputs. It only applies to Discovery and Service Mapping sensors.

The sensor runs in a scheduler worker thread. If you are tailing the node logs, you see identification_engine logging showing the identification attempts, matches, and any errors.

You can also view this logging in the syslog table by filtering on Source = identification_engine.

Solution

In this example, the error REQUIRED_ATTRIBUTE_EMPTY Missing mandatory field [owned_by] indicated that a customization made the owned_by field mandatory in the Dictionary for the cmdb table.

To resolve this issue:

Clear the Mandatory checkbox for the owned_by field in the Dictionary. Implement the mandatory requirement using a different method, such as a UI Policy, so it does not apply to the IRE. Select Run again on the ecc_queue input to create the CI and populate the ACC record.

Example 2: Invalid discovery source ACC uses the IRE Data Source/Discovery Source AgentClientCollector when matching or inserting the host server CI. ACC-V uses the value ACC-Visibility when running similar checks. You may see an identification_engine error such as:

06/14/21 13:09:30 (704) glide.scheduler.worker.1 identification_engine : INVALID_INPUT_DATA In payload invalid data source [ACC-Visibility] exist. You need to provide a valid choice value from field [discovery_source] in table [cmdb_ci]: no thrown error

For the Check run on a new ACC, the error would be For a check run on a new ACC, the error would reference AgentClientCollector instead of ACC-Visibility.

Solution

If you customized the discovery_source choice list in the past (for example, for import sets or integrations with other systems), activating the ACC apps does not add the ACC values to the choice list. ServiceNow deliberately skips customized choice lists during upgrades and plugin activations.

To resolve this issue, manually add records for the ACC values to the sys_choice table.

Related Links .ns-kb-css-body-editor-container { p { font-size: 12pt; font-family: Lato; color: #000000; } span { font-size: 12pt; font-family: Lato; color: #000000; } h2 { font-size: 24pt; font-family: Lato; color: black; } h3 { font-size: 18pt; font-family: Lato; color: black; } h4 { font-size: 14pt; font-family: Lato; color: black; } a { font-size: 12pt; font-family: Lato; color: #00718F; } a:hover { font-size: 12pt; color: #024F69; } a:target { font-size: 12pt; color: #032D42; } a:visited { font-size: 12pt; color: #00718f; } ul { font-size: 12pt; font-family: Lato; } li { font-size: 12pt; font-family: Lato; } img { display: ; max-width: ; width: ; height: ; } } Agents use the temp folder to extract or validate ACC plugins