Debugging Identification and Reconcilliation Engine using scripts in scripts background - Support and Troubleshooting

Debugging Identification and Reconcilliation Engine using scripts in scripts background Summary There are times when the outcome of Discovery/SCCM or calls to the identification engine are not as expected, i.e. when the records are not inserted/updated as anticipated. We can run Identification and Reconciliation Engine (IRE) APIs calls from background scripts to log more debug information to determine what went wrong.

Instructions WARNINGS :

The scripts in this document will perform insertion, and or update, the records. If you are debugging on a production instance, make sure you have permission to perform such insertion/update. It is best to perform the troubleshooting in a sub production instance only. We advise not to debug in production . An alternative is to use the Identification Simulation module at "Configuration > Identification/Reconciliation > Identification Simulation". However, the module will not provide the same debugging data, which may be necessary in order to get to the root of a potential issue. INFO: Generally the debug information would be available in the syslog. However if there are discovery jobs running, the data may be convoluted and hard to find the actual root cause. The debug statements from background scripts are isolated to your debugging session and therefore much simpler to analyze the output.

REQUIREMENT: IRE input payload User with admin privileges Following testing and examples performed with an admin user. Not all users will have access to scripts - background and some of the code called in these examples The IRE input payload is logged every time it is fed to the API. You can go to syslog table and search with the following filter "message" CONTAINS "input =". If the payload is not found or happened sometime time in the past, reprocess the ECC Queue input or rerun the discovery. Then search for the payload again.

Following system property may need to be added if the IRE payload is not getting logged and to see IRE debug output in the script execution:

Property name = glide.cmdb.logger.source.identification_engine Property value = info,warn,error,debug,debugVerbose Warning: Revert the property to OOB setting of info,warn,error when done with debugging.

If the IRE payload is still not found in syslog table after adding the system property then IRE API is not being used by the process inserting/updating the records.

SCRIPT: var payload = ; var discoverySource = "ServiceNow"; var output = SNC.IdentificationEngineScriptableApi.createOrUpdateCI(discoverySource, JSON.stringify(payload)); gs.print(output);

In the script above the IRE_Payload must be a JSON object (not String) in the following format {"fld1":"val1","fld2":"val2"}. Sample payload:

{"items":[{"className":"cmdb_ci_win_server","values":{"company":"86c1f3193790200044e0bfc8bcbe5d95","install_status":"1","ip_address":"10.20.30.41","location":"a63c49b037d0200044e0bfc8bcbe5dd9","mac_address":"ABCD1234","manufacturer":"0c43c22bc611227500002515e25bf079","model_id":"4431c26b37913000158bbfc8bcbe5d0d","name":"SNCTest Win Server 100","operational_status":"1","ram":"2048","serial_number":"SNC123456789"}}]}

Example Script With Payload var payload = {"items":[{"className":"cmdb_ci_win_server","values":{"company":"86c1f3193790200044e0bfc8bcbe5d95","install_status":"1","ip_address":"10.20.30.40","location":"a63c49b037d0200044e0bfc8bcbe5dd9","mac_address":"ABCD1234","manufacturer":"0c43c22bc611227500002515e25bf079","model_id":"4431c26b37913000158bbfc8bcbe5d0d","name":"SNCTest Win Server 100","operational_status":"1","ram":"2048","serial_number":"SNC123456789"}}]}; var discoverySource = "ServiceWatch"; gs.trace(true); var output = SNC.IdentificationEngineScriptableApi.createOrUpdateCI(discoverySource, JSON.stringify(payload)); gs.trace(false); gs.print(output);

Note: In the above example added gs.trace(true) to display sql statements, however not necessary in every scenario.

Example Call Output identification_engine : IdentificationEngine::process: Pass=1 identification_engine : addAttempt [{"className":"cmdb_ci_win_server","values":{"operational_status":"1","discovery_source":"ServiceWatch","install_status":"1","mac_address":"ABCD1234","name":"SNCTest002","serial_number":"SNC123456789","company":"86c1f3193790200044e0bfc8bcbe5d95","location":"a63c49b037d0200044e0bfc8bcbe5dd9","ip_address":"10.20.30.41","model_id":"4431c26b37913000158bbfc8bcbe5d0d","manufacturer":"0c43c22bc611227500002515e25bf079","ram":"2048"},"internal_id":"37dadd3adbd674104a9a53ca1184b9ac","sys_object_source_info":{"source_name":"ServiceWatch"},"settings":{},"sys_ire_info":{"ire_received_time":"2021-09-03 15:33:31"}}] - [sys_object_source] SKIPPED identification_engine : addAttempt [{"className":"cmdb_ci_win_server","values":{"operational_status":"1","discovery_source":"ServiceWatch","install_status":"1","mac_address":"ABCD1234","name":"SNCTest002","serial_number":"SNC123456789","company":"86c1f3193790200044e0bfc8bcbe5d95","location":"a63c49b037d0200044e0bfc8bcbe5dd9","ip_address":"10.20.30.41","model_id":"4431c26b37913000158bbfc8bcbe5d0d","manufacturer":"0c43c22bc611227500002515e25bf079","ram":"2048"},"internal_id":"37dadd3adbd674104a9a53ca1184b9ac","sys_object_source_info":{"source_name":"ServiceWatch"},"settings":{},"sys_ire_info":{"ire_received_time":"2021-09-03 15:33:31"}}] - [Rule id:c12f9be8c3400200d8d4bea192d3aea6|cmdb_ci_hardware|cmdb_serial_number|[serial_number, serial_number_type]] SKIPPED identification_engine : addAttempt [{"className":"cmdb_ci_win_server","values":{"operational_status":"1","discovery_source":"ServiceWatch","install_status":"1","mac_address":"ABCD1234","name":"SNCTest002","serial_number":"SNC123456789","company":"86c1f3193790200044e0bfc8bcbe5d95","location":"a63c49b037d0200044e0bfc8bcbe5dd9","ip_address":"10.20.30.41","model_id":"4431c26b37913000158bbfc8bcbe5d0d","manufacturer":"0c43c22bc611227500002515e25bf079","ram":"2048"},"internal_id":"37dadd3adbd674104a9a53ca1184b9ac","sys_object_source_info":{"source_name":"ServiceWatch"},"settings":{},"sys_ire_info":{"ire_received_time":"2021-09-03 15:33:31"}}] - [Rule id:fb27f69cc3000200d8d4bea192d3ae67|cmdb_ci_hardware|[serial_number]] MATCHED identification_engine : createOrUpdateCI: Matched 1 records and 0 relations in 4msec identification_engine : Reconciliation: update to field 'name' of CI 'a853dd72dbd674104fa1c9db1396190e' was skipped because of rules [60d9117adbd674104fa1c9db13961986] defined in cmdb_reconciliation_definition identification_engine : Reconciliation: cmdb_ci_win_server(name) field update skipped for Data Source ServiceWatch identification_engine : Commit: UPDATE cmdb_ci_win_server : a853dd72dbd674104fa1c9db1396190e identification_engine : Processed 1 records and 0 relations in 4msec + 135msec (waited 0msec for mutex) identification_engine : logId:[3bda95ba17d67410854244fa7a3ee1c2] Processed payload from ServiceWatch. Using options: {partial_payloads:false,partial_commits:false,deduplicate_payloads:false,generate_summary:false} identification_engine : logId:[3bda95ba17d67410854244fa7a3ee1c2] Input = {"items":[{"className":"cmdb_ci_win_server","values":{"operational_status":"1","install_status":"1","mac_address":"ABCD1234","name":"SNCTest002","company":"86c1f3193790200044e0bfc8bcbe5d95","location":"a63c49b037d0200044e0bfc8bcbe5dd9","serial_number":"SNC123456789","ip_address":"10.20.30.41","model_id":"4431c26b37913000158bbfc8bcbe5d0d","manufacturer":"0c43c22bc611227500002515e25bf079","ram":"2048"},"sys_object_source_info":{"source_name":"ServiceWatch"},"sys_ire_info":{"ire_received_time":"2021-09-03 15:33:31"}}]} identification_engine : logId:[3bda95ba17d67410854244fa7a3ee1c2] Output = {"items":[{"className":"cmdb_ci_win_server","operation":"UPDATE","sysId":"a853dd72dbd674104fa1c9db1396190e","maskedAttributes":["name"],"identifierEntrySysId":"fb27f69cc3000200d8d4bea192d3ae67","identificationAttempts":[{"info":"sys_object_source SKIPPED","identifierName":"","attemptResult":"SKIPPED","attributes":[],"hybridEntryCiAttributes":[]},{"identifierName":"Hardware Rule","attemptResult":"SKIPPED","attributes":["serial_number","serial_number_type"],"searchOnTable":"cmdb_serial_number","hybridEntryCiAttributes":[]},{"identifierName":"Hardware Rule","attemptResult":"MATCHED","attributes":["serial_number"],"searchOnTable":"cmdb_ci_hardware","hybridEntryCiAttributes":[]}],"info":[],"errorCount":0,"markers":[],"mergedPayloadIds":[],"warningCount":0,"inputIndices":[0]}],"additionalCommittedItems":[],"relations":[],"additionalCommittedRelations":[]} identification_engine : Identification Engine Total execution time 146msec *** Script: {"items":[{"className":"cmdb_ci_win_server","operation":"UPDATE","sysId":"a853dd72dbd674104fa1c9db1396190e","maskedAttributes":["name"],"identifierEntrySysId":"fb27f69cc3000200d8d4bea192d3ae67","identificationAttempts":[{"info":"sys_object_source SKIPPED","identifierName":"","attemptResult":"SKIPPED","attributes":[],"hybridEntryCiAttributes":[]},{"identifierName":"Hardware Rule","attemptResult":"SKIPPED","attributes":["serial_number","serial_number_type"],"searchOnTable":"cmdb_serial_number","hybridEntryCiAttributes":[]},{"identifierName":"Hardware Rule","attemptResult":"MATCHED","attributes":["serial_number"],"searchOnTable":"cmdb_ci_hardware","hybridEntryCiAttributes":[]}],"info":[],"errorCount":0,"markers":[],"mergedPayloadIds":[],"warningCount":0,"inputIndices":[0]}],"additionalCommittedItems":[],"relations":[],"additionalCommittedRelations":[],"hasError":false,"hasWarning":false} Note: In the example above the name was not updated due to a reconciliation rule.

Example 1: SCCM not updating the existing record and inserts a duplicate There is an existing Computer record with Serial Number = "ABCDE12345" so when SCCM transformation runs, it should have matched the existing Computer record. However, it didn't and created a duplicated one. Subsequent runs matched the newly created record.

After fetching the IRE Payload from syslog and run the script with gs.trace(true), we see that IRE evaluates the Identification Rules against the payload. When evaluating the Serial Number identifier, it makes the following query:

SELECT cmdb0.`sys_id`, cmdb0.`sys_mod_count`, cmdb0.`sys_class_name`, cmdb0.`serial_number` FROM (cmdb cmdb0 INNER JOIN cmdb$par1 cmdb$par10 ON cmdb0.`sys_id` = cmdb$par10.`sys_id` ) WHERE cmdb0.`sys_class_path` LIKE '/!!/!2%' AND ((cmdb0.`serial_number` = 'ABCDE12345')) AND cmdb$par10.`duplicate_of` IS NULL

Notice the condition highlighted in RED . That was the failing condition because the existing record was marked as a duplicate of another sometime in the past. The "cmdb_ci_computer"."duplicate_of" field wasn't empty so IRE couldn't find the existing record and thus a new on got created.

Example 2: SQL Pattern Not Updating CI Name in MSFT SQL Instance Class. User discovering MSSQL server using patterns that have been modified so that it populates SQL Instance name as per company requirement. However, the SQL Instance name is not getting updated after discovery runs.

Fetching the IRE Payload and running it from script background we see the following IRE Debug output.

identification_engine : Commit: NO_CHANGE cmdb_ci_win_server : fc01a2dd08224e402ccc921212fed23f identification_engine : Reconciliation: update to field 'tcp_port' of CI '6bce7dcfdb08b7c479cca9a5ca961910' was skipped because of rules [3104a82e9f503200c7445f9bc32e7054, a0d77a059311020062281f10077ffb32] defined in cmdb_reconciliation_definition identification_engine : Reconciliation: update to field 'name' of CI '6bce7dcfdb08b7c479cca9a5ca961910' was skipped because of rules [3104a82e9f503200c7445f9bc32e7054] defined in cmdb_reconciliation_definition identification_engine : Reconciliation: cmdb_ci_db_mssql_instance(tcp_port) field update skipped for Data Source Service-now identification_engine : Reconciliation: cmdb_ci_db_mssql_instance(name) field update skipped for Data Source Service-now identification_engine : Commit: UPDATE cmdb_ci_db_mssql_instance : 6bce7dcfdb08b7c479cca9a5ca961910

Here, the update to the existing SQL Instance record was aborted because there were no Reconciliation Rules for legacy "Service-now" discovery source. The relevant lines above were colored red.

Example 3: Review business rules called when the script was execute Steps

Navigate to "System Diagnostics > Script Debugger" On the Script Debugger select the "Session Log" tab Click on "Settings" and enable business rule debug Execute script from scripts - background, this will be in a separate browser tab Review session log for the business rules executed Review the script output for the IRE debug Note: Other session debug settings could be applied to see more information. Just select the checkboxes as necessary.