How are the Agent Client Collector record fields populated, and why might the CI field be empty?


Details

The Agent Client Collector (ACC) records in the instance have several status fields and a reference to the CI and MID Server. Knowing how each is populated and updated my help you resolve issues with your ACCs.

Tables

Agent Client Collectors list columns

These are the columns from the default view for a sn_agent_cmdb_ci_agent list in the instance, showing the dot-walking, and which tables fields the column values are from.

name
agent_extended_info.status
agent_extended_info.cmdb_ci --> Reference to cmdb_ci
agent_extended_info.cmdb_ci.sys_class_name
agent_extended_info.mid --> Reference to ecc_agent
ip_address
agent_extended_info.up_since
agent_extended_info.running_checks_num
agent_extended_info.data_collection
agent_extended_info.is_duplicate
agent_extended_info.last_refreshed
agent_extended_info.agent_version

Code that populates the fields

Record creation and status updates:

When an ACC is first installed the MID Server it is connected to will send a REST request to the instance. This causes the sn_agent_cmdb_ci_agent and sn_agent_ci_extended_info record pair to be created. From then on, every minute, the same REST message is sent to keep the records updated.

In the syslog_transaction table, this will look like a REST type transaction, created by the MNID Server user, with URL /api/sn_agent/agents/updateClientTimestamp?api=api

The Agent Client Collector API (/api/sn_agent/agents) is defined by this Scripted REST Web Service record:
/sys_ws_definition.do?sys_id=cf0d4208c3e3030039a3553a81d3ae9a

And the updateClientTimestamp resource is defined in:
/sys_ws_operation.do?sys_id=b64dc208c3e3030039a3553a81d3ae76

That in turn makes use of Script Include MonitoringConfig, and it's functions updateClients and updateMidUpdateTime:
/sys_script_include.do?sys_id=6c62a3f0c305130039a3553a81d3ae88

If the ACC is new, insertNewClients function will insert the pair of records. Otherwise the function updateClientKeepAlive will update the values.

sn_agent_cmdb_ci_agent fields updated:

sn_agent_ci_extended_info fields updated:

Host CI identification and CI reference:

The ACC will then need to discovery itself in order to either create a new Host Server CI in the CMDB, or link to an existing CI record.

The "check-discovery-basic" Check is used to do this, using command check_discover.rb
/sn_agent_check_def.do?sys_id=f9fe1e3c536113006dfeddeeff7b12a2

In Paris/ACC 2.2 that check definition record states plugin asset "osquery" is required.

A probe output is placed in the ECC Queue for the MID Server to run, which in turn will cause the ACC to discover itself. (This is for initial discovery. Depending on customer setup there might not be an output payload, only inputs)

agent: MID Server of the ACC
topic: MonitoringProbe
name: on_demand_request
source: on_demand_request
payload: e.g.:

{
  "clients_cis": {
    "ff9ada72acd05615": {}
  },
  "checks": [
    {
      "name": "check-discovery-basic",
      "command": "check_discover.rb",
      "interval": "43200",
      "cron": "",
      "timeout": 180,
      "id": "f9fe1e3c536113006dfeddeeff7b12a2",
      "assets": "osquery",
      "check_type_id": "4048774567633300b7b72dbd2685efa1",
      "mid_script": "AgentDiscoveryResultParser",
      "instance_script": "true",
      "priority": 2,
      "requestId": "bb70d032dba364d056aed3f3ca96196e"
    }
  ]
}

The input payload will contain a JSON object that includes the ACC Host's full details, including serial number and other identifying info. e.g.

[
  {
    "client": "MSEDGEWIN10",
    "agent_id": "ff9ada72acd05615",
    "check": {
      "command": "check_discover.rb",
      "name": "check-discovery-basic",
      "interval": 43200,
      "timeout": 180,
    "output": "CheckDiscover OK: {\"hostname\":\"MSEDGEWIN10\",\"serial\":\"012345\",\"processes\":[{\"Name\":\"System Idle Process\",\"Caption\":\"System Idle Process\",\...

The Sensor for that input is the AgentNowResponseProcessor business rule, which uses script include AgentNowHandler function processEccRecord.

As the Check is Discovery type (/sn_agent_check_type.do?sys_id=4048774567633300b7b72dbd2685efa1) the DiscoveryHandler script include handleDiscovery and createHostCis functions will create or update the host CI using the normal IRE APIs, and find the ACC record and update the cmdb_ci field on it.

UPDATE: In Quebec/ACC-F 2.3.0 and above, endpoint_discovery.rb is used instead of check_discover.rb by both ACC-F and ACC-V. You will see other assets are required, in addition to osquery: acc-f-commons, acc-f-modules. This may also change in future. 

Troubleshooting

As with any check that is dependent on plugin assets, in order to run that check, the ACC will first need to download the plugin. In the case of Paris/ACC 2.2, "osquery" (/sn_agent_asset_name.do?sys_id=175dddb4533a1010b9ffddeeff7b1203), and in the related records of that the actual OS-specific tarballs are attached, and the URL the ACC will be requesting that from can be worked out, e.g.:

http://<mid server IP>:<web service port>/static/acc_plugin/linux/all/all/all/osquery.tar.gz 

If for any reason that file cannot be downloaded, such as security policies, the plugins are not synced to the agent yet, so the check will not execute, which will then cause the probe to fail, and the Host CI will not be linked to the Agent record. That will then prevent you doing nearly everything with this ACC, as Monitoring events/metrics can't be tied to the CI in the CMDB.

The acc.log file, with debug turned on in the acc.yml configuration file, may show error such as this in that situation:

{"component":"agent","error":"error getting plugin for check: sha512 'f60522ecfc7db6d18f50b6e317905e8e4bb737757c0b4b7642a53af19ca7aef3fcb349af36c643a216ab240d5f069797800f8ea6ba074207e3ff7854acbc1f5f' does not match specified sha512: 'd7f242d5bccefbac4c5cc0068aaef416a96424d6503393b5e45ca706f8fc1a3b924324f28f32140b95ad1ffbce0f8722dd0e2c54be38d5ab3c6675b339099e95'","level":"error","msg":"sending failure","time":"2021-04-07T04:01:54+01:00"}

This is misleading because the SAH512 check is comparing the contents of the 'file', with the SHA256 for this particular plugin asset saved in the asset.db file. In this particular case, it was comparing the expected SHA256 checksum with an html error page from a network device that had blocked the request for the file, and not the actual file at all.

{"level":"info","msg":"finished fetching asset","status":"503 Service Unavailable","time":"2021-04-07T03:59:55+01:00"}

This is also misleading, because the MID Web server was not having any problems at the time. However both are clues that the ACC couldn't get the osquery.tar.gz  file, so didn't have the plugin asset, and couldn't run the discovery.

To troubleshoot this, cURL (or wget) can be used to replicate the request from the ACC host. e.g.

[root@acchost ~]# curl -Lk -u web_service_username:password http://10.x.x.x:8081/static/acc_plugin/linux/all/all/all/osquery.tar.gz
<html>
<head>
<title>File Transfer Blocked</title>
...
<h1>File Transfer Blocked</h1>
<p>Transfer of the file you were trying to download or upload has been blocked in accordance with company policy. Please contact your system administrator if you believe this is in error.</p>
<p><b>File name:</b> osquery.tar.gz </p>
...

In this situation, exceptions will need making in whatever software policy or firewall/proxy is blocking the file type, to solve the issues. 

The file is downloaded via the temp folder of the ACC host (or temp folder of the user the ACC service runs as), so any restrictions/permissions issues for these file types in that folder may also need investigating. The plugin asset is staged in temp while further security checks are made, it is then extracted to the appropriate cache folder for the agent, and the temp files deleted.

Check 'output' is not the expected JSON IRE payload:

PRB1509426 causes the following result value in the ecc queue input for the endpoint_discovery.rb check. This is fixed by upgrading to ACC-F 2.5.0 or later.

"output" : "C:/Program Files/ServiceNow/agent-client-collector/embedded/lib/ruby/gems/2.7.0/gems/json-2.3.0/lib/json/common.rb:156:in `parse': 783: unexpected token at '' (JSON::ParserError)\r\n\tfrom C:/Program Files/ServiceNow/agent-client-collector/embedded/lib/ruby/gems/2.7.0/gems/json-2.3.0/lib/json/common.rb:156:in `parse'\r\n\tfrom C:/ProgramData/ServiceNow/agent-client-collector/cache/acc-f-commons/bin/endpoint_discovery.rb:179:in `block in &lt;main&gt;'\r\n\tfrom C:/ProgramData/ServiceNow/agent-client-collector/cache/acc-f-commons/bin/endpoint_discovery.rb:166:in `each'\r\n\tfrom C:/ProgramData/ServiceNow/agent-client-collector/cache/acc-f-commons/bin/endpoint_discovery.rb:166:in `&lt;main&gt;'\r\n",

Other script errors may be seen, which may indicate other problems.

IRE Insert failed Example 1

This example is a ACC-F 2.5.0, on Quebec Patch 5, where the MID Server and ACC are on a Server 2019 lab instance.

Note that Data Collection is off. Clicking "Resume data collection" on the agent form won't help with this.

Find the ecc_queue input for MonitoringProbe/on_demand_request where the input payload contains "name" : "check-discovery-basic". The ecc_queue payload field should include a large JSON output, that has all the details of the ACC host's server. e.g.

"output" : "{\"basic_inventory\":{\"platform\":\"windows\",\"name\":\"cs-win8695\",\"manufacturer\":\"VMware, Inc.\",\"model_id\":\"VMware Virtual Platform\",\"model_number\":\"\",\"serial_number\":\"VMware-42 34 fd 2e 71 c2 80 c9-57 3c 4e e8 5e e9 d6 3f\",\"operating_system_domain\":\"supportlabwin.local\",\"operating_system\":\"Microsoft Windows Server 2019 Standard\",\"operating_system_version\":\"10.0.17763\",\"operating_system_service_pack\":\"\",\"operating_system_architecture\":\"64-bit\",\"os_address_width\":\"64\",\"ram\":\"4096\",\"cpu_manufacturer\":\"GenuineIntel\",\"cpu_name\":\"Intel(R) Xeon(R) Gold 5120 CPU @ 2.20GHz\",\"cpu_speed\":\"2195\",\"cpu_count\":\"1\",\"cpu_core_count\":\"1\",\"cpu_core_thread\":\"1\",\"kernel_release\":\"\",\"ip_address\":\"10.190.132.12\",\"default_gateway\":\"10.190.132.1\",\"is_virtual\":\"true\",\"fully_qualified_domain_name\":\"cs-win8695.supportlabwin.local\",\"object_id\":\"\",\"start_date\":\"1626852162000\",\"short_description\":\"\"},\"tcp_connections\":{\"connections\":[{\"type\":\"on\",\"pid\":\"860\",\"ip\":\"0.0.0.0\",\"port\":\"135\"},{\"type\":\"on\",\"pid\":\"4\",\"ip\":\"0.0.0.0\",\"port\":\"445\"},{\"type\":\"on\",\"pid\":\"404\",\"ip\":\"0.0.0.0\",\"port\":\"3389\"},{\"....

If you don't have the attributes of the host in that output, then something will have gone wrong when running the check in the ACC, so you will need the ACC logs.

However if you do have the data like in this example, then the problem is likely to be due to the CMDB Identification & Reconciliation engine having a problem inserting the CIs.

You can (temporarily!) add the IRE debug system property glide.cmdb.logger.source.identification_engine = info,warn,error,debug,debugVerbose, to turn on full IRE debug. Then click "Run again" on the ecc_queue input form, to force the sensor to re-run.

Note: "Run again (debug)", where you can then use Session Debug, won't work for ACC inputs, as that only applies to Discovery/Service Mapping sensors. 

The sensor runs in a scheduler worker thread, so if you are tailing the node logs, you will see identification_engine logging. e.g.

07/23/21 13:15:58 (752) glide.scheduler.worker.6 identification_engine                      : isDependency: checking if sn_agent_cmdb_ci_agent is a dependency for cmdb_ci_win_server
07/23/21 13:15:58 (753) glide.scheduler.worker.6 identification_engine                      : isDependency: looking through hosting for cmdb_ci_win_server
07/23/21 13:15:58 (755) glide.scheduler.worker.6 identification_engine                      : isDependency: RESULT: sn_agent_cmdb_ci_agent is *not* a dependency for cmdb_ci_win_server
07/23/21 13:15:58 (756) glide.scheduler.worker.6 identification_engine                      : isDependency: checking if cmdb_ci_win_server is a dependency for sn_agent_cmdb_ci_agent
07/23/21 13:15:58 (756) glide.scheduler.worker.6 identification_engine                      : isDependency: looking through hosting for sn_agent_cmdb_ci_agent
07/23/21 13:15:58 (757) glide.scheduler.worker.6 identification_engine                      : isDependency: RESULT: cmdb_ci_win_server is *not* a dependency for sn_agent_cmdb_ci_agent
07/23/21 13:15:58 (760) glide.scheduler.worker.6 identification_engine                      : IdentificationEngine::process: Pass=1
07/23/21 13:15:58 (761) glide.scheduler.worker.6 identification_engine                      : addAttempt [{"className":"cmdb_ci_win_server","values":{"os_address_width":"64","cpu_name":"Intel(R) Xeon(R) Gold 5120 CPU @ 2.20GHz","default_gateway":"10.190.132.1","short_description":"","virtual":"true","fqdn":"cs-win8695.supportlabwin.local","os_domain":"supportlabwin.local","cpu_speed":"2195","cpu_core_thread":"1","serial_number":"VMware-42 34 fd 2e 71 c2 80 c9-57 3c 4e e8 5e e9 d6 3f","kernel_release":"","ip_address":"10.190.132.12","object_id":"","operating_system_service_pack":"","cpu_count":"1","discovery_source":"Manual Entry","operating_system_architecture":"64-bit","name":"cs-win8695","model_number":"","host_name":"cs-win8695","cpu_core_count":"1","ram":"4096"},"internal_id":"312dc059c9a57850c5afddf553f865ba","sys_object_source_info":{"source_name":"Manual Entry"},"settings":{},"sys_ire_info":{"ire_received_time":"2021-07-23 11:15:58"}}] - [sys_object_source] SKIPPED
07/23/21 13:15:58 (762) glide.scheduler.worker.6 identification_engine                      : addAttempt [{"className":"cmdb_ci_win_server","values":{"os_address_width":"64","cpu_name":"Intel(R) Xeon(R) Gold 5120 CPU @ 2.20GHz","default_gateway":"10.190.132.1","short_description":"","virtual":"true","fqdn":"cs-win8695.supportlabwin.local","os_domain":"supportlabwin.local","cpu_speed":"2195","cpu_core_thread":"1","serial_number":"VMware-42 34 fd 2e 71 c2 80 c9-57 3c 4e e8 5e e9 d6 3f","kernel_release":"","ip_address":"10.190.132.12","object_id":"","operating_system_service_pack":"","cpu_count":"1","discovery_source":"Manual Entry","operating_system_architecture":"64-bit","name":"cs-win8695","model_number":"","host_name":"cs-win8695","cpu_core_count":"1","ram":"4096"},"internal_id":"312dc059c9a57850c5afddf553f865ba","sys_object_source_info":{"source_name":"Manual Entry"},"settings":{},"sys_ire_info":{"ire_received_time":"2021-07-23 11:15:58"}}] - [Rule id:c12f9be8c3400200d8d4bea192d3aea6|cmdb_ci_hardware|cmdb_serial_number|[serial_number, serial_number_type]] SKIPPED
07/23/21 13:15:58 (764) glide.scheduler.worker.6 identification_engine                      : addAttempt [{"className":"cmdb_ci_win_server","values":{"os_address_width":"64","cpu_name":"Intel(R) Xeon(R) Gold 5120 CPU @ 2.20GHz","default_gateway":"10.190.132.1","short_description":"","virtual":"true","fqdn":"cs-win8695.supportlabwin.local","os_domain":"supportlabwin.local","cpu_speed":"2195","cpu_core_thread":"1","serial_number":"VMware-42 34 fd 2e 71 c2 80 c9-57 3c 4e e8 5e e9 d6 3f","kernel_release":"","ip_address":"10.190.132.12","object_id":"","operating_system_service_pack":"","cpu_count":"1","discovery_source":"Manual Entry","operating_system_architecture":"64-bit","name":"cs-win8695","model_number":"","host_name":"cs-win8695","cpu_core_count":"1","ram":"4096"},"internal_id":"312dc059c9a57850c5afddf553f865ba","sys_object_source_info":{"source_name":"Manual Entry"},"settings":{},"sys_ire_info":{"ire_received_time":"2021-07-23 11:15:58"}}] - [Rule id:fb27f69cc3000200d8d4bea192d3ae67|cmdb_ci_hardware|[serial_number]] NO_MATCH
07/23/21 13:15:58 (766) glide.scheduler.worker.6 identification_engine                      : addAttempt [{"className":"cmdb_ci_win_server","values":{"os_address_width":"64","cpu_name":"Intel(R) Xeon(R) Gold 5120 CPU @ 2.20GHz","default_gateway":"10.190.132.1","short_description":"","virtual":"true","fqdn":"cs-win8695.supportlabwin.local","os_domain":"supportlabwin.local","cpu_speed":"2195","cpu_core_thread":"1","serial_number":"VMware-42 34 fd 2e 71 c2 80 c9-57 3c 4e e8 5e e9 d6 3f","kernel_release":"","ip_address":"10.190.132.12","object_id":"","operating_system_service_pack":"","cpu_count":"1","discovery_source":"Manual Entry","operating_system_architecture":"64-bit","name":"cs-win8695","model_number":"","host_name":"cs-win8695","cpu_core_count":"1","ram":"4096"},"internal_id":"312dc059c9a57850c5afddf553f865ba","sys_object_source_info":{"source_name":"Manual Entry"},"settings":{},"sys_ire_info":{"ire_received_time":"2021-07-23 11:15:58"}}] - [Rule id:556eb250c3400200d8d4bea192d3ae92|cmdb_ci_hardware|[name]] NO_MATCH
07/23/21 13:15:58 (767) glide.scheduler.worker.6 identification_engine                      : addAttempt [{"className":"cmdb_ci_win_server","values":{"os_address_width":"64","cpu_name":"Intel(R) Xeon(R) Gold 5120 CPU @ 2.20GHz","default_gateway":"10.190.132.1","short_description":"","virtual":"true","fqdn":"cs-win8695.supportlabwin.local","os_domain":"supportlabwin.local","cpu_speed":"2195","cpu_core_thread":"1","serial_number":"VMware-42 34 fd 2e 71 c2 80 c9-57 3c 4e e8 5e e9 d6 3f","kernel_release":"","ip_address":"10.190.132.12","object_id":"","operating_system_service_pack":"","cpu_count":"1","discovery_source":"Manual Entry","operating_system_architecture":"64-bit","name":"cs-win8695","model_number":"","host_name":"cs-win8695","cpu_core_count":"1","ram":"4096"},"internal_id":"312dc059c9a57850c5afddf553f865ba","sys_object_source_info":{"source_name":"Manual Entry"},"settings":{},"sys_ire_info":{"ire_received_time":"2021-07-23 11:15:58"}}] - [Rule id:8f476e29c3440200d8d4bea192d3ae0f|cmdb_ci_hardware|cmdb_ci_network_adapter|[mac_address, name]] SKIPPED
07/23/21 13:15:58 (768) glide.scheduler.worker.6 identification_engine                      : REQUIRED_ATTRIBUTE_EMPTY Missing mandatory field [owned_by] in table [cmdb_ci_win_server]. Add input value for mandatory field in payload: no thrown error
07/23/21 13:15:58 (769) glide.scheduler.worker.6 identification_engine                      : addAttempt [{"className":"sn_agent_cmdb_ci_agent","values":{"discovery_source":"Manual Entry","agent_id":"f1bffb00b06f6f2b"},"internal_id":"f12dc059a0a57850c32835e2cb15e2ba","sys_object_source_info":{"source_name":"Manual Entry"},"settings":{},"sys_ire_info":{"ire_received_time":"2021-07-23 11:15:58"}}] - [sys_object_source] SKIPPED
07/23/21 13:15:58 (771) glide.scheduler.worker.6 identification_engine                      : addAttempt [{"className":"sn_agent_cmdb_ci_agent","values":{"discovery_source":"Manual Entry","agent_id":"f1bffb00b06f6f2b"},"internal_id":"f12dc059a0a57850c32835e2cb15e2ba","sys_object_source_info":{"source_name":"Manual Entry"},"settings":{},"sys_ire_info":{"ire_received_time":"2021-07-23 11:15:58"}}] - [Rule id:5bd061d553511010c360ddeeff7b1251|sn_agent_cmdb_ci_agent|[agent_id]] MATCHED
07/23/21 13:15:58 (771) glide.scheduler.worker.6 identification_engine                      : IdentificationEngine::process: Pass=2
07/23/21 13:15:58 (772) glide.scheduler.worker.6 identification_engine                      : ABANDONED Too many other errors: no thrown error
07/23/21 13:15:58 (773) glide.scheduler.worker.6 identification_engine                      : ABANDONED Too many other errors: no thrown error
07/23/21 13:15:58 (775) glide.scheduler.worker.6 identification_engine                      : identifyCI: Matched 2 records and 1 relations in 0msec
07/23/21 13:15:58 (804) glide.scheduler.worker.6 identification_engine                      : logId:[712dc059dba578506f1a9d62ba9619bc] Detected error while processing payload from Manual Entry. Using options: {partial_payloads:false,partial_commits:false,deduplicate_payloads:false,generate_summary:false}: no thrown error
07/23/21 13:15:58 (806) glide.scheduler.worker.6 identification_engine                      : logId:[712dc059dba578506f1a9d62ba9619bc] Input = {"items":[{"className":"cmdb_ci_win_server","values":{"os_address_width":"64","cpu_name":"Intel(R) Xeon(R) Gold 5120 CPU @ 2.20GHz","default_gateway":"10.190.132.1","short_description":"","virtual":"true","fqdn":"cs-win8695.supportlabwin.local","os_domain":"supportlabwin.local","cpu_speed":"2195","cpu_core_thread":"1","serial_number":"VMware-42 34 fd 2e 71 c2 80 c9-57 3c 4e e8 5e e9 d6 3f","kernel_release":"","ip_address":"10.190.132.12","object_id":"","operating_system_service_pack":"","cpu_count":"1","operating_system_architecture":"64-bit","name":"cs-win8695","model_number":"","host_name":"cs-win8695","cpu_core_count":"1","ram":"4096"},"sys_object_source_info":{"source_name":"Manual Entry"},"sys_ire_info":{"ire_received_time":"2021-07-23 11:15:58"}},{"className":"sn_agent_cmdb_ci_agent","values":{"agent_id":"f1bffb00b06f6f2b"},"sys_object_source_info":{"source_name":"Manual Entry"},"sys_ire_info":{"ire_received_time":"2021-07-23 11:15:58"}}],"relations":[{"parent":1,"child":0,"type":"Runs on::Runs","sys_rel_source_info":{"source_feed":"UNKNOWN","source_name":"Manual Entry"}}]}: no thrown error
07/23/21 13:15:58 (808) glide.scheduler.worker.6 identification_engine                      : logId:[712dc059dba578506f1a9d62ba9619bc] Output = {"items":[{"className":"cmdb_ci_win_server","sysId":"Unknown","identifierEntrySysId":"Unknown","errors":[{"error":"REQUIRED_ATTRIBUTE_EMPTY","message":"Missing mandatory field [owned_by] in table [cmdb_ci_win_server]. Add input value for mandatory field in payload"},{"error":"ABANDONED","message":"Too many other errors"}],"identificationAttempts":[{"info":"sys_object_source SKIPPED","attemptResult":"SKIPPED","identifierName":"","attributes":[],"hybridEntryCiAttributes":[]},{"attemptResult":"SKIPPED","identifierName":"Hardware Rule","attributes":["serial_number","serial_number_type"],"hybridEntryCiAttributes":[],"searchOnTable":"cmdb_serial_number"},{"attemptResult":"NO_MATCH","identifierName":"Hardware Rule","attributes":["serial_number"],"hybridEntryCiAttributes":[],"searchOnTable":"cmdb_ci_hardware"},{"attemptResult":"NO_MATCH","identifierName":"Hardware Rule","attributes":["name"],"hybridEntryCiAttributes":[],"searchOnTable":"cmdb_ci_hardware"},{"attemptResult":"SKIPPED","identifierName":"Hardware Rule","attributes":["mac_address","name"],"hybridEntryCiAttributes":[],"searchOnTable":"cmdb_ci_network_adapter"}],"info":[],"errorCount":2,"inputIndices":[0],"mergedPayloadIds":[],"warningCount":0,"markers":[]},{"className":"sn_agent_cmdb_ci_agent","operation":"UPDATE","sysId":"0c078851dba578506f1a9d62ba961900","identifierEntrySysId":"5bd061d553511010c360ddeeff7b1251","identificationAttempts":[{"info":"sys_object_source SKIPPED","attemptResult":"SKIPPED","identifierName":"","attributes":[],"hybridEntryCiAttributes":[]},{"attemptResult":"MATCHED","identifierName":"Agent Client Collector","attributes":["agent_id"],"hybridEntryCiAttributes":[],"searchOnTable":"sn_agent_cmdb_ci_agent"}],"info":[],"errorCount":0,"inputIndices":[1],"mergedPayloadIds":[],"warningCount":0,"markers":[]}],"additionalCommittedItems":[],"relations":[{"className":"cmdb_rel_ci","sysId":"Unknown","identifierEntrySysId":"Unknown","errors":[{"error":"ABANDONED","message":"Too many other errors"}],"errorCount":1,"inputIndices":[0],"mergedPayloadIds":[],"warningCount":0,"markers":[]}],"additionalCommittedRelations":[]}: no thrown error
07/23/21 13:15:58 (809) glide.scheduler.worker.6 identification_engine                      : Identification Engine Total execution time 64msec
07/23/21 13:15:58 (888) glide.scheduler.worker.6 Completed: ASYNC: AgentNowResponseProcessor in 0:00:00.288, next occurrence is null

You can also see the same logging in the syslog table, if you filter on Source = identification_engine

Solution:

In this particular example, "REQUIRED_ATTRIBUTE_EMPTY Missing mandatory field [owned_by]" indicated that a customisation had made the owned_by field of the the cmdb table Mandatory in the Dictionary.

The solution was to untick mandatory in the Dictionary, and implement that mandatory requirement a different way, perhaps using a UI Policy, so that it wouldn't also apply to the IRE.

Re-running the ecc_queue input again, allowed the CI to be created and the information to be added to the ACC record:

IRE Insert failed Example 2

ACC will use IRE Data Source/Discovery Source "AgentClientCollector" when matching/inserting the host server CI.  ACC-V uses value "ACC-Visibility", when running it's similar Checks. You may see an identification_engine error in relation to that:

06/14/21 13:09:30 (704) glide.scheduler.worker.1 identification_engine : INVALID_INPUT_DATA In payload invalid data source [ACC-Visibility] exist. You need to provide a valid choice value from field [discovery_source] in table [cmdb_ci]: no thrown error 

For the Check run on a new ACC, the error would be "identification_engine : INVALID_INPUT_DATA In payload invalid data source [AgentClientCollector] exist. You need to provide a valid choice value from field [discovery_source] in table [cmdb_ci]: no thrown error "

Solution:

If you have customised the discovery_source choice list in the past to add your own values, perhaps for import sets, or integrations with other systems, then the activation of the ACC apps will not have added these to the choice list, because we deliberately skip customised choice lists in upgrades and plugin activations.

Records for the ACC values may need manually adding to the sys_choice table: