How are the Agent Client Collector record fields populated, and why might the CI field be empty? - Support and Troubleshooting

How are the Agent Client Collector record fields populated, and why might the CI field be empty? Details The Agent Client Collector (ACC) records in the instance have several status fields and a reference to the CI and MID Server. Knowing how each is populated and updated my help you resolve issues with your ACCs.

Tables Agent Client Collectors [sn_agent_cmdb_ci_agent] The main record. This is a CMDB CI Class. If you are referencing an ACC from another record, this is the one you reference. Agent Client Collector Info [sn_agent_ci_extended_info] Additional fields. Each of the above records has one of these too, and references it. Most of the status columns seen on a list of Agents are actually dot-walked from this record. MID Server [ecc_agent] The referenced MID Server record, which is currently the MID Server that the ACC is in communication with the instance via. Configuration Item The referenced host server that the ACC is installed on Agent Client Collectors list columns These are the columns from the default view for a sn_agent_cmdb_ci_agent list in the instance, showing the dot-walking, and which tables fields the column values are from.

name agent_extended_info. status agent_extended_info. cmdb_ci --> Reference to cmdb_ci agent_extended_info.cmdb_ci. sys_class_name agent_extended_info. mid --> Reference to ecc_agent ip_address agent_extended_info. up_since agent_extended_info. running_checks_num agent_extended_info. data_collection agent_extended_info. is_duplicate agent_extended_info. last_refreshed agent_extended_info. agent_version

Code that populates the fields Record creation and status updates: When an ACC is first installed, the MID Server it is connected to will send a REST request to the instance. This causes the sn_agent_cmdb_ci_agent and sn_agent_ci_extended_info record pair to be created. From then on, every minute, the same REST message is sent to keep the records updated.

In the syslog_transaction table, this will look like a REST type transaction, created by the MID Server user, with URL /api/sn_agent/agents/ updateClientTimestamp ?api=api

The Agent Client Collector API (/api/sn_agent/agents) is defined by this Scripted REST Web Service record: /sys_ws_definition.do?sys_id=cf0d4208c3e3030039a3553a81d3ae9a

And the updateClientTimestamp resource is defined in: /sys_ws_operation.do?sys_id=b64dc208c3e3030039a3553a81d3ae76

That in turn makes use of Script Include MonitoringConfig, and it's functions updateClients and classifyClientsAndUpdate. /sys_script_include.do?sys_id=6c62a3f0c305130039a3553a81d3ae88

This API handles 3 situations:

Newly installed ACCs Full update Update timestamp and status only. If the ACC is new, insertNewClients function will insert the pair of records. Otherwise the function updateClientKeepAlive will update the values.

sn_agent_cmdb_ci_agent fields updated:

name agent_id ip_address sn_agent_ci_extended_info fields updated:

status mid up_since name agent_id ip_address is_windows agent_version Note: The "Delete agent info" business rule ensures no "orphans" are left on sn_agent_ci_extended_info by deleting the corresponding sn_agent_ci_extended_info when a sn_agent_cmdb_ci_agent record is deleted. When this rule is inactive, "orphan" sn_agent_ci_extended_info records are kept which can lead to the empty sn_agent_cmdb_ci_agent records being created. This can happen if, as one example, an agent is deleted from the table however is still installed (the next keep alive will find the "orphan" sn_agent_ci_extended_info record and create an empty sn_agent_cmdb_ci_agent record).

Host CI identification and CI reference: The ACC will then need to discovery itself in order to either create a new Host Server CI in the CMDB, or link to an existing CI record.

The "check-discovery-basic" Check is used to do this, using command check_discover.rb /sn_agent_check_def.do?sys_id=f9fe1e3c536113006dfeddeeff7b12a2

In Paris/ACC 2.2 that check definition record states plugin asset "osquery" is required.

The hourly scheduled job "Refresh host data for Agents", will run for all agents that have an empty Host reference, or have not been re-discovered for more time than set in system property "sn_agent.host_data_collection_refresh_duration_seconds", which defaults to 43200 (12 hours). /sysauto_script.do?sys_id=e59cc3d2531030100112ddeeff7b12f0

If you don't want to wait for that job to run, you can click "Collect Host Data" button on the agent form.

A probe output is placed in the ECC Queue for the MID Server to run, which in turn will cause the ACC to discover itself. (This is for initial discovery. Depending on customer setup there might not be an output payload, only inputs)

agent: MID Server of the ACC topic: MonitoringProbe name: on_demand_request source: on_demand_request payload: e.g.:

{ "clients_cis": { "ff9ada72acd05615": {} }, "checks": [ { "name": " check-discovery-basic ", "command": " check_discover.rb ", "interval": "43200", "cron": "", "timeout": 180, "id": "f9fe1e3c536113006dfeddeeff7b12a2", "assets": "osquery", "check_type_id": "4048774567633300b7b72dbd2685efa1", "mid_script": "AgentDiscoveryResultParser", "instance_script": "true", "priority": 2, "requestId": "bb70d032dba364d056aed3f3ca96196e" } ] } The input payload will contain a JSON object that includes the ACC Host's full details, including serial number and other identifying info. e.g.

[ { "client": "MSEDGEWIN10", "agent_id": "ff9ada72acd05615", "check": { "command": "check_discover.rb", "name": "check-discovery-basic", "interval": 43200, "timeout": 180, "output": "CheckDiscover OK: {\" hostname\":\"MSEDGEWIN10\",\"serial\":\"012345 \",\"processes\":[{\"Name\":\"System Idle Process\",\"Caption\":\"System Idle Process\",\... The Sensor for that input is the AgentNowResponseProcessor business rule, which uses script include AgentNowHandler function processEccRecord.

As the Check is Discovery type (/sn_agent_check_type.do?sys_id=4048774567633300b7b72dbd2685efa1) the DiscoveryHandler script include handleDiscovery and createHostCis functions will create or update the host CI using the normal IRE APIs , and find the ACC record and update the cmdb_ci field on it.

UPDATE: In Quebec/ACC-F 2.3.0 and above, endpoint_discovery.rb is used instead of check_discover.rb by both ACC-F and ACC-V. You will see other assets are required, in addition to osquery: acc-f-commons, acc-f-modules. This may also change in future.

Troubleshooting As with any check that is dependent on plugin assets, in order to run that check, the ACC will first need to download the plugin. In the case of Paris/ACC 2.2, "osquery" (/sn_agent_asset_name.do?sys_id=175dddb4533a1010b9ffddeeff7b1203), and in the related records of that the actual OS-specific tarballs are attached, and the URL the ACC will be requesting that from can be worked out, e.g.:

https:// : /static/acc_plugin/linux/all/all/all/osquery.tar.gz If for any reason that file cannot be downloaded, such as security policies, the plugins are not synced to the agent yet, so the check will not execute, which will then cause the probe to fail, and the Host CI will not be linked to the Agent record. That will then prevent you doing nearly everything with this ACC, as Monitoring events/metrics can't be tied to the CI in the CMDB.

The acc.log file, with debug turned on in the acc.yml configuration file, may show error such as this in that situation:

{"component":"agent","error":"error getting plugin for check: sha512 'f60522ecfc7db6d18f50b6e317905e8e4bb737757c0b4b7642a53af19ca7aef3fcb349af36c643a216ab240d5f069797800f8ea6ba074207e3ff7854acbc1f5f' does not match specified sha512: 'd7f242d5bccefbac4c5cc0068aaef416a96424d6503393b5e45ca706f8fc1a3b924324f28f32140b95ad1ffbce0f8722dd0e2c54be38d5ab3c6675b339099e95'","level":"error","msg":"sending failure","time":"2021-04-07T04:01:54+01:00"} This is misleading because the SAH512 check is comparing the contents of the 'file', with the SHA256 for this particular plugin asset saved in the asset.db file. In this particular case, it was comparing the expected SHA256 checksum with an html error page from a network device that had blocked the request for the file, and not the actual file at all.

{"level":"info","msg":"finished fetching asset","status":"503 Service Unavailable","time":"2021-04-07T03:59:55+01:00"}

This is also misleading, because the MID Web server was not having any problems at the time. However both are clues that the ACC couldn't get the osquery.tar.gz file, so didn't have the plugin asset, and couldn't run the discovery.

To troubleshoot this, cURL (or wget) can be used to replicate the request from the ACC host. e.g.

[root@acchost ~]# curl -Lk -u web_service_username:password https://10.x.x.x:8081/static/acc_plugin/linux/all/all/all/osquery.tar.gz File Transfer Blocked ... File Transfer Blocked Transfer of the file you were trying to download or upload has been blocked in accordance with company policy. Please contact your system administrator if you believe this is in error.

File name: osquery.tar.gz

... In this situation, exceptions will need making in whatever software policy or firewall/proxy is blocking the file type, to solve the issues.

The file is downloaded via the temp folder of the ACC host (or temp folder of the user the ACC service runs as - see KB1640697 for details ), so any restrictions/permissions issues for these file types in that folder may also need investigating. The plugin asset is staged in temp while further security checks are made, it is then extracted to the appropriate cache folder for the agent, and the temp files deleted.

Check 'output' is not the expected JSON IRE payload: PRB1509426 causes the following result value in the ecc queue input for the endpoint_discovery.rb check. This is fixed by upgrading to ACC-F 2.5.0 or later.

"output" : "C:/Program Files/ServiceNow/agent-client-collector/embedded/lib/ruby/gems/2.7.0/gems/json-2.3.0/lib/json/common.rb:156:in `parse': 783: unexpected token at '' (JSON::ParserError)\r\n\tfrom C:/Program Files/ServiceNow/agent-client-collector/embedded/lib/ruby/gems/2.7.0/gems/json-2.3.0/lib/json/common.rb:156:in `parse'\r\n\tfrom C:/ProgramData/ServiceNow/agent-client-collector/cache/acc-f-commons/bin/endpoint_discovery.rb:179:in `block in &lt;main&gt;'\r\n\tfrom C:/ProgramData/ServiceNow/agent-client-collector/cache/acc-f-commons/bin/endpoint_discovery.rb:166:in `each'\r\n\tfrom C:/ProgramData/ServiceNow/agent-client-collector/cache/acc-f-commons/bin/endpoint_discovery.rb:166:in `&lt;main&gt;'\r\n", Other script errors may be seen, which may indicate other problems.

IRE Insert failed Example 1

This example is a ACC-F 2.5.0, on Quebec Patch 5, where the MID Server and ACC are on a Server 2019 lab instance.

Note that Data Collection is off. Clicking "Resume data collection" on the agent form won't help with this.

Find the ecc_queue input for MonitoringProbe/on_demand_request where the input payload contains "name" : "check-discovery-basic". The ecc_queue payload field should include a large JSON output, that has all the details of the ACC host's server. e.g.

"output" : "{\"basic_inventory\":{\"platform\":\"windows\",\"name\":\"cs-win8695\",\"manufacturer\":\"VMware, Inc.\",\"model_id\":\"VMware Virtual Platform\",\"model_number\":\"\",\"serial_number\":\"VMware-42 34 fd 2e 71 c2 80 c9-57 3c 4e e8 5e e9 d6 3f\",\"operating_system_domain\":\"supportlabwin.local\",\"operating_system\":\"Microsoft Windows Server 2019 Standard\",\"operating_system_version\":\"10.0.17763\",\"operating_system_service_pack\":\"\",\"operating_system_architecture\":\"64-bit\",\"os_address_width\":\"64\",\"ram\":\"4096\",\"cpu_manufacturer\":\"GenuineIntel\",\"cpu_name\":\"Intel(R) Xeon(R) Gold 5120 CPU @ 2.20GHz\",\"cpu_speed\":\"2195\",\"cpu_count\":\"1\",\"cpu_core_count\":\"1\",\"cpu_core_thread\":\"1\",\"kernel_release\":\"\",\"ip_address\":\"10.190.132.12\",\"default_gateway\":\"10.190.132.1\",\"is_virtual\":\"true\",\"fully_qualified_domain_name\":\"cs-win8695.supportlabwin.local\",\"object_id\":\"\",\"start_date\":\"1626852162000\",\"short_description\":\"\"},\"tcp_connections\":{\"connections\":[{\"type\":\"on\",\"pid\":\"860\",\"ip\":\"0.0.0.0\",\"port\":\"135\"},{\"type\":\"on\",\"pid\":\"4\",\"ip\":\"0.0.0.0\",\"port\":\"445\"},{\"type\":\"on\",\"pid\":\"404\",\"ip\":\"0.0.0.0\",\"port\":\"3389\"},{\".... If you don't have the attributes of the host in that output, then something will have gone wrong when running the check in the ACC, so you will need the ACC logs.

However if you do have the data like in this example, then the problem is likely to be due to the CMDB Identification & Reconciliation engine having a problem inserting the CIs.

You can (temporarily!) add the IRE debug system property glide.cmdb.logger.source.identification_engine = info,warn,error,debug,debugVerbose , to turn on full IRE debug. Then click "Run again" on the ecc_queue input form, to force the sensor to re-run.

Note: "Run again (debug)", where you can then use Session Debug, won't work for ACC inputs, as that only applies to Discovery/Service Mapping sensors.

The sensor runs in a scheduler worker thread, so if you are tailing the node logs, you will see identification_engine logging. e.g.

07/23/21 13:15:58 (752) glide.scheduler.worker.6 identification_engine : isDependency: checking if sn_agent_cmdb_ci_agent is a dependency for cmdb_ci_win_server 07/23/21 13:15:58 (753) glide.scheduler.worker.6 identification_engine : isDependency: looking through hosting for cmdb_ci_win_server 07/23/21 13:15:58 (755) glide.scheduler.worker.6 identification_engine : isDependency: RESULT: sn_agent_cmdb_ci_agent is *not* a dependency for cmdb_ci_win_server 07/23/21 13:15:58 (756) glide.scheduler.worker.6 identification_engine : isDependency: checking if cmdb_ci_win_server is a dependency for sn_agent_cmdb_ci_agent 07/23/21 13:15:58 (756) glide.scheduler.worker.6 identification_engine : isDependency: looking through hosting for sn_agent_cmdb_ci_agent 07/23/21 13:15:58 (757) glide.scheduler.worker.6 identification_engine : isDependency: RESULT: cmdb_ci_win_server is *not* a dependency for sn_agent_cmdb_ci_agent 07/23/21 13:15:58 (760) glide.scheduler.worker.6 identification_engine : IdentificationEngine::process: Pass=1 07/23/21 13:15:58 (761) glide.scheduler.worker.6 identification_engine : addAttempt [{"className":"cmdb_ci_win_server","values":{"os_address_width":"64","cpu_name":"Intel(R) Xeon(R) Gold 5120 CPU @ 2.20GHz","default_gateway":"10.190.132.1","short_description":"","virtual":"true","fqdn":"cs-win8695.supportlabwin.local","os_domain":"supportlabwin.local","cpu_speed":"2195","cpu_core_thread":"1","serial_number":"VMware-42 34 fd 2e 71 c2 80 c9-57 3c 4e e8 5e e9 d6 3f","kernel_release":"","ip_address":"10.190.132.12","object_id":"","operating_system_service_pack":"","cpu_count":"1","discovery_source":"Manual Entry","operating_system_architecture":"64-bit","name":"cs-win8695","model_number":"","host_name":"cs-win8695","cpu_core_count":"1","ram":"4096"},"internal_id":"312dc059c9a57850c5afddf553f865ba","sys_object_source_info":{"source_name":"Manual Entry"},"settings":{},"sys_ire_info":{"ire_received_time":"2021-07-23 11:15:58"}}] - [sys_object_source] SKIPPED 07/23/21 13:15:58 (762) glide.scheduler.worker.6 identification_engine : addAttempt [{"className":"cmdb_ci_win_server","values":{"os_address_width":"64","cpu_name":"Intel(R) Xeon(R) Gold 5120 CPU @ 2.20GHz","default_gateway":"10.190.132.1","short_description":"","virtual":"true","fqdn":"cs-win8695.supportlabwin.local","os_domain":"supportlabwin.local","cpu_speed":"2195","cpu_core_thread":"1","serial_number":"VMware-42 34 fd 2e 71 c2 80 c9-57 3c 4e e8 5e e9 d6 3f","kernel_release":"","ip_address":"10.190.132.12","object_id":"","operating_system_service_pack":"","cpu_count":"1","discovery_source":"Manual Entry","operating_system_architecture":"64-bit","name":"cs-win8695","model_number":"","host_name":"cs-win8695","cpu_core_count":"1","ram":"4096"},"internal_id":"312dc059c9a57850c5afddf553f865ba","sys_object_source_info":{"source_name":"Manual Entry"},"settings":{},"sys_ire_info":{"ire_received_time":"2021-07-23 11:15:58"}}] - [Rule id:c12f9be8c3400200d8d4bea192d3aea6|cmdb_ci_hardware|cmdb_serial_number|[serial_number, serial_number_type]] SKIPPED 07/23/21 13:15:58 (764) glide.scheduler.worker.6 identification_engine : addAttempt [{"className":"cmdb_ci_win_server","values":{"os_address_width":"64","cpu_name":"Intel(R) Xeon(R) Gold 5120 CPU @ 2.20GHz","default_gateway":"10.190.132.1","short_description":"","virtual":"true","fqdn":"cs-win8695.supportlabwin.local","os_domain":"supportlabwin.local","cpu_speed":"2195","cpu_core_thread":"1","serial_number":"VMware-42 34 fd 2e 71 c2 80 c9-57 3c 4e e8 5e e9 d6 3f","kernel_release":"","ip_address":"10.190.132.12","object_id":"","operating_system_service_pack":"","cpu_count":"1","discovery_source":"Manual Entry","operating_system_architecture":"64-bit","name":"cs-win8695","model_number":"","host_name":"cs-win8695","cpu_core_count":"1","ram":"4096"},"internal_id":"312dc059c9a57850c5afddf553f865ba","sys_object_source_info":{"source_name":"Manual Entry"},"settings":{},"sys_ire_info":{"ire_received_time":"2021-07-23 11:15:58"}}] - [Rule id:fb27f69cc3000200d8d4bea192d3ae67|cmdb_ci_hardware|[serial_number]] NO_MATCH 07/23/21 13:15:58 (766) glide.scheduler.worker.6 identification_engine : addAttempt [{"className":"cmdb_ci_win_server","values":{"os_address_width":"64","cpu_name":"Intel(R) Xeon(R) Gold 5120 CPU @ 2.20GHz","default_gateway":"10.190.132.1","short_description":"","virtual":"true","fqdn":"cs-win8695.supportlabwin.local","os_domain":"supportlabwin.local","cpu_speed":"2195","cpu_core_thread":"1","serial_number":"VMware-42 34 fd 2e 71 c2 80 c9-57 3c 4e e8 5e e9 d6 3f","kernel_release":"","ip_address":"10.190.132.12","object_id":"","operating_system_service_pack":"","cpu_count":"1","discovery_source":"Manual Entry","operating_system_architecture":"64-bit","name":"cs-win8695","model_number":"","host_name":"cs-win8695","cpu_core_count":"1","ram":"4096"},"internal_id":"312dc059c9a57850c5afddf553f865ba","sys_object_source_info":{"source_name":"Manual Entry"},"settings":{},"sys_ire_info":{"ire_received_time":"2021-07-23 11:15:58"}}] - [Rule id:556eb250c3400200d8d4bea192d3ae92|cmdb_ci_hardware|[name]] NO_MATCH 07/23/21 13:15:58 (767) glide.scheduler.worker.6 identification_engine : addAttempt [{"className":"cmdb_ci_win_server","values":{"os_address_width":"64","cpu_name":"Intel(R) Xeon(R) Gold 5120 CPU @ 2.20GHz","default_gateway":"10.190.132.1","short_description":"","virtual":"true","fqdn":"cs-win8695.supportlabwin.local","os_domain":"supportlabwin.local","cpu_speed":"2195","cpu_core_thread":"1","serial_number":"VMware-42 34 fd 2e 71 c2 80 c9-57 3c 4e e8 5e e9 d6 3f","kernel_release":"","ip_address":"10.190.132.12","object_id":"","operating_system_service_pack":"","cpu_count":"1","discovery_source":"Manual Entry","operating_system_architecture":"64-bit","name":"cs-win8695","model_number":"","host_name":"cs-win8695","cpu_core_count":"1","ram":"4096"},"internal_id":"312dc059c9a57850c5afddf553f865ba","sys_object_source_info":{"source_name":"Manual Entry"},"settings":{},"sys_ire_info":{"ire_received_time":"2021-07-23 11:15:58"}}] - [Rule id:8f476e29c3440200d8d4bea192d3ae0f|cmdb_ci_hardware|cmdb_ci_network_adapter|[mac_address, name]] SKIPPED 07/23/21 13:15:58 (768) glide.scheduler.worker.6 identification_engine : REQUIRED_ATTRIBUTE_EMPTY Missing mandatory field [owned_by] in table [cmdb_ci_win_server]. Add input value for mandatory field in payload: no thrown error 07/23/21 13:15:58 (769) glide.scheduler.worker.6 identification_engine : addAttempt [{"className":"sn_agent_cmdb_ci_agent","values":{"discovery_source":"Manual Entry","agent_id":"f1bffb00b06f6f2b"},"internal_id":"f12dc059a0a57850c32835e2cb15e2ba","sys_object_source_info":{"source_name":"Manual Entry"},"settings":{},"sys_ire_info":{"ire_received_time":"2021-07-23 11:15:58"}}] - [sys_object_source] SKIPPED 07/23/21 13:15:58 (771) glide.scheduler.worker.6 identification_engine : addAttempt [{"className":"sn_agent_cmdb_ci_agent","values":{"discovery_source":"Manual Entry","agent_id":"f1bffb00b06f6f2b"},"internal_id":"f12dc059a0a57850c32835e2cb15e2ba","sys_object_source_info":{"source_name":"Manual Entry"},"settings":{},"sys_ire_info":{"ire_received_time":"2021-07-23 11:15:58"}}] - [Rule id:5bd061d553511010c360ddeeff7b1251|sn_agent_cmdb_ci_agent|[agent_id]] MATCHED 07/23/21 13:15:58 (771) glide.scheduler.worker.6 identification_engine : IdentificationEngine::process: Pass=2 07/23/21 13:15:58 (772) glide.scheduler.worker.6 identification_engine : ABANDONED Too many other errors: no thrown error 07/23/21 13:15:58 (773) glide.scheduler.worker.6 identification_engine : ABANDONED Too many other errors: no thrown error 07/23/21 13:15:58 (775) glide.scheduler.worker.6 identification_engine : identifyCI: Matched 2 records and 1 relations in 0msec 07/23/21 13:15:58 (804) glide.scheduler.worker.6 identification_engine : logId:[712dc059dba578506f1a9d62ba9619bc] Detected error while processing payload from Manual Entry. Using options: {partial_payloads:false,partial_commits:false,deduplicate_payloads:false,generate_summary:false}: no thrown error 07/23/21 13:15:58 (806) glide.scheduler.worker.6 identification_engine : logId:[712dc059dba578506f1a9d62ba9619bc] Input = {"items":[{"className":"cmdb_ci_win_server","values":{"os_address_width":"64","cpu_name":"Intel(R) Xeon(R) Gold 5120 CPU @ 2.20GHz","default_gateway":"10.190.132.1","short_description":"","virtual":"true","fqdn":"cs-win8695.supportlabwin.local","os_domain":"supportlabwin.local","cpu_speed":"2195","cpu_core_thread":"1","serial_number":"VMware-42 34 fd 2e 71 c2 80 c9-57 3c 4e e8 5e e9 d6 3f","kernel_release":"","ip_address":"10.190.132.12","object_id":"","operating_system_service_pack":"","cpu_count":"1","operating_system_architecture":"64-bit","name":"cs-win8695","model_number":"","host_name":"cs-win8695","cpu_core_count":"1","ram":"4096"},"sys_object_source_info":{"source_name":"Manual Entry"},"sys_ire_info":{"ire_received_time":"2021-07-23 11:15:58"}},{"className":"sn_agent_cmdb_ci_agent","values":{"agent_id":"f1bffb00b06f6f2b"},"sys_object_source_info":{"source_name":"Manual Entry"},"sys_ire_info":{"ire_received_time":"2021-07-23 11:15:58"}}],"relations":[{"parent":1,"child":0,"type":"Runs on::Runs","sys_rel_source_info":{"source_feed":"UNKNOWN","source_name":"Manual Entry"}}]}: no thrown error 07/23/21 13:15:58 (808) glide.scheduler.worker.6 identification_engine : logId:[712dc059dba578506f1a9d62ba9619bc] Output = {"items":[{"className":"cmdb_ci_win_server","sysId":"Unknown","identifierEntrySysId":"Unknown","errors":[{"error":"REQUIRED_ATTRIBUTE_EMPTY","message":"Missing mandatory field [owned_by] in table [cmdb_ci_win_server]. Add input value for mandatory field in payload"},{"error":"ABANDONED","message":"Too many other errors"}],"identificationAttempts":[{"info":"sys_object_source SKIPPED","attemptResult":"SKIPPED","identifierName":"","attributes":[],"hybridEntryCiAttributes":[]},{"attemptResult":"SKIPPED","identifierName":"Hardware Rule","attributes":["serial_number","serial_number_type"],"hybridEntryCiAttributes":[],"searchOnTable":"cmdb_serial_number"},{"attemptResult":"NO_MATCH","identifierName":"Hardware Rule","attributes":["serial_number"],"hybridEntryCiAttributes":[],"searchOnTable":"cmdb_ci_hardware"},{"attemptResult":"NO_MATCH","identifierName":"Hardware Rule","attributes":["name"],"hybridEntryCiAttributes":[],"searchOnTable":"cmdb_ci_hardware"},{"attemptResult":"SKIPPED","identifierName":"Hardware Rule","attributes":["mac_address","name"],"hybridEntryCiAttributes":[],"searchOnTable":"cmdb_ci_network_adapter"}],"info":[],"errorCount":2,"inputIndices":[0],"mergedPayloadIds":[],"warningCount":0,"markers":[]},{"className":"sn_agent_cmdb_ci_agent","operation":"UPDATE","sysId":"0c078851dba578506f1a9d62ba961900","identifierEntrySysId":"5bd061d553511010c360ddeeff7b1251","identificationAttempts":[{"info":"sys_object_source SKIPPED","attemptResult":"SKIPPED","identifierName":"","attributes":[],"hybridEntryCiAttributes":[]},{"attemptResult":"MATCHED","identifierName":"Agent Client Collector","attributes":["agent_id"],"hybridEntryCiAttributes":[],"searchOnTable":"sn_agent_cmdb_ci_agent"}],"info":[],"errorCount":0,"inputIndices":[1],"mergedPayloadIds":[],"warningCount":0,"markers":[]}],"additionalCommittedItems":[],"relations":[{"className":"cmdb_rel_ci","sysId":"Unknown","identifierEntrySysId":"Unknown","errors":[{"error":"ABANDONED","message":"Too many other errors"}],"errorCount":1,"inputIndices":[0],"mergedPayloadIds":[],"warningCount":0,"markers":[]}],"additionalCommittedRelations":[]}: no thrown error 07/23/21 13:15:58 (809) glide.scheduler.worker.6 identification_engine : Identification Engine Total execution time 64msec 07/23/21 13:15:58 (888) glide.scheduler.worker.6 Completed: ASYNC: AgentNowResponseProcessor in 0:00:00.288, next occurrence is null You can also see the same logging in the syslog table, if you filter on Source = identification_engine

Solution:

In this particular example, "REQUIRED_ATTRIBUTE_EMPTY Missing mandatory field [owned_by]" indicated that a customisation had made the owned_by field of the the cmdb table Mandatory in the Dictionary.

The solution was to untick mandatory in the Dictionary, and implement that mandatory requirement a different way, perhaps using a UI Policy, so that it wouldn't also apply to the IRE.

Re-running the ecc_queue input again, allowed the CI to be created and the information to be added to the ACC record:

IRE Insert failed Example 2 ACC will use IRE Data Source/Discovery Source "AgentClientCollector" when matching/inserting the host server CI. ACC-V uses value "ACC-Visibility", when running it's similar Checks. You may see an identification_engine error in relation to that:

06/14/21 13:09:30 (704) glide.scheduler.worker.1 identification_engine : INVALID_INPUT_DATA In payload invalid data source [ACC-Visibility] exist. You need to provide a valid choice value from field [discovery_source] in table [cmdb_ci]: no thrown error For the Check run on a new ACC, the error would be "identification_engine : INVALID_INPUT_DATA In payload invalid data source [AgentClientCollector] exist . You need to provide a valid choice value from field [discovery_source] in table [cmdb_ci]: no thrown error "

Solution:

If you have customised the discovery_source choice list in the past to add your own values, perhaps for import sets, or integrations with other systems, then the activation of the ACC apps will not have added these to the choice list, because we deliberately skip customised choice lists in upgrades and plugin activations.

Records for the ACC values may need manually adding to the sys_choice table: