CIs Processed Via IRE Get Well Playbook Configuration Items without a 'discovery_source' value A guide for how to manage CIs with empty (unpopulated) discovery source values Table of Contents Summary Audience Before you begin Problem Overview Executive Summary How this playbook can help you achieve your business goals How this playbook is structured Problem Analysis Upstream Causes Downstream Consequences 4 Impact on Your Business 5 Engagement Questions 6 Remediation Plays Summary Play 1: Analysis 7 Play 2: Fix Play: Use IH-ETL to import Custom Data Sources Play 3: Fix Play: Add IRE to any active Discovery Sources Play 4: Data Governance Play Summary Goal of this Playbook The goal of this playbook is to help you identify and manage those Configuration Items (CIs) that don't have values in the discovery source field (discovery_source). A valid entry in the discovery source field is required if you want to use the Identification and Reconciliation Engine (IRE) to maintain the integrity of your data. Details about this playbook Author Michael Walker, Callan BondDate 05/01/2025Addresses HSD # HSD0003671, HSD0006970Applicable ServiceNow Releases All ReleasesTime required Approximately 4 to 20 hours Before you begin HSD0001855 on discovery_source choices should be completed prior to this playbook to ensure all Discovery sources are set up correctly in the system Problem Overview Discovery identifies the source of the data of a CI. Specially, it uses the value in the discovery source (discovery_ source) field in the CI record. With this value, the platform or application can use the rules of the IRE. These rules provide fine-tuned control over multiple sources of discovered data. If the discovery source field is empty (unpopulated), Discovery can't identify the source of the data in the CI. This means you won't be able to take full advantage of the IRE to maintain the accuracy your data. This playbook helps you identify the CIs without discovery source values, and tells you how to add these values to the CIs. Executive Summary How this playbook can help you achieve your business goals The IRE helps you maintain the integrity of your data in the Configuration Management Database (CMDB). How this playbook is structured This playbook guides you through through five plays. Play 1 (an analysis play) helps you find the CIs in your CMDB that have empty (unpopulated) discovery source values. Play 2 (one of the three fix plays) tells you how to add the missing discovery source value by using the Update Multiple feature.Play 3 (the second of the three fix plays) explains the script you can use to bulk update the CIs.Play 4 (the last of the three fix plays) shows you how to add scripted calls to the IRE to frequently used Transform Maps.Play 5 (a Data Governance play) lists the guidelines and processes for CI identification and reconciliation. Use these guidelines to maintain the health of the CMDB. Problem Analysis The scriptable IRE or CMDBTransformUtil wasn't available when you first started using the ServiceNow platform, and you haven't been able to overhaul your legacy integrations.The third-party integration vendors haven't adopted the IRE.You may not be aware that your manual processes and data imports can be processed through IRE. Downstream Consequences Data Consequence If the wrong CI is chosen to match and update, someone else's CI just vanished, overwritten by the data from another Config ItemUpdating the wrong CI is can inadvertently overwrite the data belonging to someone elseIncorrectly matching CIs can create duplicate CIs. Operational Consequence Erodes trust in the accuracy of the CMDB. Results in a poor user-experienceProcesses that rely on the accuracy of the CI data in the CMDB are compromisedYou might need to create special queries to filter data around duplicates. Creating these queries increases overhead costsRemediating duplicate CIs takes valuable time away from running your business. Application Consequence Other ServiceNow products depend on the accuracy what's in the CI record. These products include Change Management, Request Management, Incident Management, Vulnerability Response, and Application Portfolio Management.Asset Management suffers due to duplicate Assets or CIs. Duplicate CIs can also create bad links. Incident Management and Alerts don't bind correctly if there are duplicate records matching the Event Management data.Change Requests are only reliable during an audit if the CMDB data is reliable.Change automation / groupings can fail when server information is duplicated within the CMDB.Service Mapping and links to Application Portfolio can fail whenever there are duplicate CIs. Only the correct one is linked to a Business Offering / Application.Service Mapping topology updates take longer or may fail completely when CIs are inaccurate or not unique. Impact on Your Business Inaccurate CI records can adversely affect the following areas of your business. Lower MTTR Data Completeness Incident Reduction Data Accuracy Increase Operational Visibility Lifecycle Management Audit/Compliance System Integrity Process Automation Data Accuracy Better User Experience User Confidence Most users will notice poor data quality and take away a poor impression of the entire platform without ever seriously reporting the issue. Getting back a lost first impression is difficult if not impossible Engagement Questions Consider the answers to these questions: What are the major sources for the CI data in your CMDB?Have you integrated any third-party Discovery Sources into your CMDB? If so, do these integrations use the CMDBTransformUtil / IdentifyAndReconcile REST API?Do you import data manually? If so, what process do you use? Does your process include the IRE API in the transforms? Remediation Plays Summary Play Play 1: Analysis What this play is about This play helps you find the CIs in your CMDB that have empty (unpopulated) discovery source values. Required tasks Use the provided script to find these CIs. Play 2: Use IH-ETL to import Custom Data Sources What this play is about This play helps to move from legacy approach to IH-ETL for data import Required tasks Follow the steps provided in this play Play 3: Add IRE to any active Discovery Sources What this play is about Explains how you can add scripted calls to the IRE to frequently used Transform Maps. Required tasks Follow the steps to create the scripts. Play 1: Analysis What this play is about In this task you will learn more about the types of CIs that have not been brought in via the IRE. When the IRE processes data to insert into the CMDB for the first time, it creates a record in the table sys_object_source. This record contains a unique identifier about the data as it exists in the source system, and a unique identifier (sys_id) of the CI that is created in the CMDB. This allows the IRE to match source data to CMDB data in the future. To determine if a CI in the CMDB has already been brought the IRE checks to see if a corresponding record for the CI exists in sys_object_source. When the IRE creates or updates a CI, it always populates the Discovery Source column of the CI. One common indicator that a record has not been processed via the IRE is that the Discovery Source field is blank. Required tasks The Data Foundations dashboards look at two main classes to determine the score for this metric: cmdb_ci_hardware, and cmdb_ci_vm_instance. Due to potential performance issues the Dashboards do not show a detailed list of CIs that did not come in via the IRE since this can be well into the millions. To check your current state of IRE usage for hardware and VM CIs, use the CMDB and CSDM Data Foundations Dashboards. Step 1: If you are on Xanadu family release or higher, the CMDB and CSDM Data Foundations Dashboards are available out of box. If you are on a version previous to Xanadu, navigate to the ServiceNow Store, search for “CMDB Data Foundations”, and install the free Store app. View the ServiceNow docs site to learn more about installing and configuring the Dashboards if you are just setting them up for the first time. Step 2: Navigate to the CMDB Data Foundation Dashboard Step 3: Navigate to the “Data Management Practices” tab and review the “CIs Processed via IRE” metric. Step 4: If your score is 100% no further action is needed. If lower, you may wish to further analyze CIs that are not being brought in via the IRE. Detailed Analysis Start with analysis of the Hardware (cmdb_ci_hardware) class. You can repeat most of these steps for VM Instance (cmdb_ci_vm_instance). Step 1: In the navigation bar, search for the term “cmdb_ci_hardware.filter”. This will bring up a list view of cmdb_ci_hardware, but not display any records as first you will apply a filter. Step 2: Set a filter to only look at CIs with no Discovery Source, and Operational Status of Operational (or whatever Lifecycle Stage field and value your organization uses to denote an active CI) Step 3: From the list controls in the top left, Group By “Class”. Step 4: At this point you will need to do further analysis on each class. Often CIs that were not brought in via the IRE are legacy data imported when the CMDB was first created, either through a direct data load or other manual means. You can add a condition to your filter to look at the Updated field, which will tell you the last time the CI was touched. Check for Updated of > 60 days or > 90 days. It is recommended that operational data (e.g., hardware or VMs) in the CMDB be kept current. Normally, operational CIs are discovered through automated means every day (or week at the latest), so the Updated date is typically 1 – 7 days old. If data is stale, it should be remediated (e.g., Discovery schedules created or fixed) or Retired (use Data Manager to retire). Refer to KB0829106 for more information on managing Stale CI data. Play 2: Fix Play: Use IH-ETL to import Custom Data Sources What this play is about All modern ServiceNow integration methods leverage the Identification and Reconciliation Engine (IRE). If you have current CMDB data that is being updated (Updated date within the last 90 days), but Discovery Source is blank, it is likely that the integration was done using a legacy method. Required tasks Common third-party data source If the integration is to a common packaged third-party vendor, there is a good chance there is an existing Service Graph Connector (SGC) that can be used to integrate and import your data. Service Graph Connectors are certified integrations that bring in data from third-party systems into the CMDB in the correct manner and via the IRE. You can find Service Graph Connectors on the ServiceNow Store. The Store listings contain links to the appropriate docs page for the Connector and guidance on installation. Niche third-party source or custom source If the integration is to a niche third-party vendor or to a custom developed application, there is likely not a prepackaged Service Graph Connector that can be used. In this case, IntegrationHub ETL (IH-ETL) should be used to develop the integration. By using IH-ETL, you are using an approved method for connecting and importing data, and this data will pass through the IRE before being inserted into the CMDB. The IRE will populate Discovery Source, minimize duplicate data, and perform other checks and reconciliations to ensure your CMDB has high quality data. Creating an IH-ETL integration is beyond the scope of this KB, but aside from guidance in the ServiceNow docs site, there is a comprehensive five-part series on YouTube. Creating a CMDB Integration tutorial using IH-ETL: Part 1 of 5 Part 2, Part 3, Part 4, Part 5 Play 3: Fix Play: Add IRE to active Discovery Sources Required tasks If you have an existing manual import using Transform maps for third party data, you can use this play to add a call to the IRE. Consider retiring any processes that use the legacy Load Data feature. Transition to using a Transform Map with IRE. Navigate to Transform Maps. Enter "*computer" into the Target table search box and press enter. The resulting maps should be good candidates for using the IRE. Note: SCCM 2016 Computer Identity should be patched using the process documented in the Community here. The following step may need to be performed within the production environment with the help of a ServiceNow customer administrator. No changes are being performed; This step shows which Import Sets are using Transform Maps targeting the CMDB. 3. Navigate to System Import Sets > Advanced > Transform History 4. Use the Filter to select Transform Map ==> Table Transform Map fields 5. Select Target table - contains - cmdb_ci 6. Click Run. The resulting list should show candidate Transform Maps that should be using the IRE, also known as CMDBTransformUtil. 7. Navigate to System Import Sets > Administration > Transform Maps and open up the map you need to update with IRE. 8. In a separate Browser Tab Navigate to the CMDBTransformUtil documentation. Locate the example, or copy it from here: // add this code to the onBefore transform map script // Call CMDB API to do Identification and Reconciliation of current row var cmdbUtil = new CMDBTransformUtil(); cmdbUtil.setDataSource('ImportSet'); cmdbUtil.identifyAndReconcile(source, map, log); ignore = true; if (cmdbUtil.hasError()) { var errorMessage = cmdbUtil.getError(); log.error(errorMessage); } else { log.info('IE Output Payload: ' + cmdbUtil.getOutputPayload()); log.info('Imported CI: ' + cmdbUtil.getOutputRecordSysId()); target.get(cmdbUtil.getOutputRecordSysId()); //the above line is not in the web example, but super helpful } 9. On your Transform Map, scroll to the Related Lists, and click New under "Transform Scripts." 10. Change the new record's When drop-down to "onBefore" 11. Paste the contents of the example into the script section, replacing the existing comment. Your window should look like this: 12. Click Submit to save the record If you open the record to review your work, it will similar to the following: 13. Test the import / transform process end-to-end. This is to ensure the addition of the IRE / CMDBTransformUtil is working as expected. Also a comment will be present on each row stating that Row Transform was ignored by Script. This is as expected. The work to insert / update records has been handled in the onBefore script, while the Transform itself has ignored our rows of data. The Target Record field should still be useful when checking your work. (Thanks to optional line 16 in our example above) Play 4: Data Governance Play What this play is about This play lists the best practices and processes for CI identification and reconciliation Required tasks Periodically review the data in your CMDB and run the Analysis play again to check if there are CIs that were added to the system without going through the IRE. Congratulations You have completed this Get Well Playbook.