[CMDB - IRE] How the CMDB Identification and Reconciliation Engine works when passing a CI (as payload) to the createOrUpdateCi() - Support and Troubleshooting

[CMDB - IRE] How the CMDB Identification and Reconciliation Engine works when passing a CI (as payload) to the createOrUpdateCi() Issue This article explains how exactly the Identification and Reconciliation Engine works when a CI (as payload) is being sent to the IRE. This article explains the below items:

Table of Contents How does the Identification and Reconciliation Engine Work? Functionality Details 1# Data Sources 2# Payload Validation 3# CI Identification based on Payload 4# Process the CI Matches 5# Lookup for related or dependents 6# Apply the Reconciliation 7# Commit the data to the database 8# Commit the data to the database Troubleshooting How does the Identification and Reconciliation Engine Work? Identification and Reconciliation Engine (IRE) working functionality diagram:

Functionality Details 1# Data Sources Here we get all the data from different data sources like SCCM, Discovery, Service Mapping, Import Sets, different integrations like Qualys, etc. Using these data sources we build the JSON payload and make advantage of CMDB API call to createOrUpdateCi() 2# Payload Validation In this phase, once we send the data to the IRE engine the very first step would be to validate the payload like if the incoming payload is well-formatted, are there any errors if any required attributes are missing, etc. 3# CI Identification based on Payload 3.1# Payload with no Errors If the payload has no errors then, the system will look for the matching CMDB identifier rules (cmdb_identifier). (First Independent rules of that particular class we are sending, parent class identification based on configuration, etc. currently, most of the devices fall under "Hardware Rule" on cmdb_ci_hardware). Once the matching identification rule is identified then the engine will look for the identifier entries within that rule (on a priority basis until a match is found). 3.2# Payload with Errors Based on the payload validation, if the payload has any errors then it will stop processing and throw errors to the "System Logs". The detailed CMDB IRE logs can be obtained by enabling a few properties.

4# Process the CI Matches Once the IRE finds the matching CIs, then the below steps are taken accordingly based on no of matches.

4.1# No Matches Found (=0) If no matches found then the payload is reserved for creating/insertion of a new CI.

4.2# Exactly one CI Found (=1) If exactly one match found then the IRE will pick that CI and reserve it for Update of that CI. Now it will look for any LookUp/related/reference rules defined for that CI type or its parent CI type. (Both insert/update) Then the system will check for the Reconciliation Rules/Data Precedence rules for the CI and its attributes and then it will update the CI accordingly. 4.3# More than one CI Matches Found (>1 but less than the skip_duplicates.threshold) This is a special case as the system found multiple CIs matched based on the identification rules. Once the IRE finds multiple CIs matching then immediately it will generate the De-Duplication task . Once the duplicate task is created then it will try to process the duplicate CIs. Processing of these duplicates is performed in different ways Before Madrid and after Madrid. Till Madrid, the duplicate CIs are marked as Duplicate by setting the discovery_source as Duplicate. New York, onwards we do not use the discovery_source instead, we have a new field duplicate_of which will be populated with main CI for all duplicates. Details are as below: 4.3.1# Skip the Duplicates If the system is configured not to process the duplicates (By setting the property glide.identification_engine.skip_duplicates to false OR if the maximum threshold (glide.identification_engine.skip_duplicates.threshold - default 5) is less than the number of duplicates found then IRE will not process the CIs.

4.3.2# Process the Duplicates: If property glide.identification_engine.skip_duplicates is true and the number of duplicates is less than the maximum threshold (glide.identification_engine.skip_duplicates.threshold) then the IRE will mark the CIs as Duplicate (until Madrid release) or will not populate the duplicate_of field (from New York release). 5# Lookup for related or dependents Based on the configuration and the payload the IRE will try to check/insert the lookup, related, and relationship data that needs to be inserted or updated.

6# Apply the Reconciliation If the Ci needs to be updated, then we will try to apply the reconciliation rules and data precedence rules. The reason we do this only for the update because for a new CI (insert) we don't need to check all these as it's a new CI.

7# Commit the data to the database At this point, we commit the data to the database and get sys_id.

8# Commit the data to the database Here the processed data/errors results will be returned to the API caller.

Troubleshooting How to run the CI identification on-demand using the payload How to check the data/payload being sent to IRE and check the issues with the payload Identification engine error messages Duplicate CMDB CI records Debugging Identification and Reconcilliation Engine using scripts in scripts background Related Links CMDB identifier rules Identification engine error messages