CIs Processed Via IRE - Data Foundations

CIs Processed Via IRE  Get Well Playbook

Configuration Items without a 'discovery_source' value

A guide for how to manage CIs with empty (unpopulated) discovery source values

Table of Contents

Summary

Audience

Before you begin

Problem Overview

Executive Summary

How this playbook can help you achieve your business goals

How this playbook is structured

Problem Analysis

Upstream Causes

Downstream Consequences 4

Impact on Your Business 5

Engagement Questions 6

Remediation Plays

Summary

Play 1: Analysis 7

Play 2: Fix Play: Replace the missing values using the Update Multiple feature

Play 3: Fix Play: (Optional) Set the discovery_source field in bulk

Play 4: Fix Play: Add IRE to any active Discovery Sources

Data Governance

Summary

Goal of this Playbook

The goal of this playbook is to help you identify and manage those Configuration Items (CIs) that don't have values in the discovery source field (discovery_source). A valid entry in the discovery source field is required if you want to use the Identification and Reconciliation Engine (IRE) to maintain the integrity of your data.

Details about this playbook

Author John (Johnny) Walker Date 05/01/2020 Addresses HSD # HSD0003671, HSD0006970 Applicable ServiceNow Releases Madrid, New York Time required Approximately 4 to 20 hours

Before you begin

HSD0001855 on discovery_source choices should be completed prior to this playbook to ensure all Discovery sources are set up correctly in the system

Problem Overview

Discovery identifies the source of the data of a CI. Specially, it uses the value in the discovery source (discovery_ source) field in the CI record.

With this value, the platform or application can use the rules of the IRE. These rules provide fine-tuned control over multiple sources of discovered data.

If the discovery source field is empty (unpopulated), Discovery can't identify the source of the data in the CI. This means you won't be able to take full advantage of the IRE to maintain the accuracy your data.

This playbook helps you identify the CIs without discovery source values, and tells you how to add these values to the CIs.

Executive Summary

How this playbook can help you achieve your business goals

The IRE helps you maintain the integrity of your data in the Configuration Management Database (CMDB).

How this playbook is structured

This playbook guides you through through five plays.

Play 1 (an analysis play) helps you find the CIs in your CMDB that have empty (unpopulated) discovery source values.

Play 2 (one of the three fix plays) tells you how to add the missing discovery source value by using the Update Multiple feature.

Play 3 (the second of the three fix plays) explains the script you can use to bulk update the CIs.

Play 4 (the last of the three fix plays) shows you how to add scripted calls to the IRE to frequently used Transform Maps.

Play 5 (a Data Governance play) lists the guidelines and processes for CI identification and reconciliation. Use these guidelines to maintain the health of the CMDB.

Problem Analysis

The scriptable IRE or CMDBTransformUtil wasn't available when you first started using the ServiceNow platform, and you haven't been able to overhaul your legacy integrations.

The third-party integration vendors haven't adopted the IRE.

You may not be aware that your manual processes and data imports can be processed through IRE.

Downstream Consequences

Data Consequence

If the wrong CI is chosen to match and update, someone else's CI just vanished, overwritten by the data from another Config Item

Updating the wrong CI is can inadvertently overwrite the data belonging to someone else

Incorrectly matching CIs can create duplicate CIs.

Operational Consequence

Erodes trust in the accuracy of the CMDB. Results in a poor user-experience

Processes that rely on the accuracy of the CI data in the CMDB are compromised

You might need to create special queries to filter data around duplicates. Creating these queries increases overhead costs

Remediating duplicate CIs takes valuable time away from running your business.

Application Consequence

Other ServiceNow products depend on the accuracy what's in the CI record.

These products include Change Management, Request Management, Incident Management, Vulnerability Response, and Application Portfolio Management.

Asset Management suffers due to duplicate Assets or CIs. Duplicate CIs can also create bad links.

Incident Management and Alerts don't bind correctly if there are duplicate records matching the Event Management data.

Change Requests are only reliable during an audit if the CMDB data is reliable.

Change automation / groupings can fail when server information is duplicated within the CMDB.

Service Mapping and links to Application Portfolio can fail whenever there are duplicate CIs. Only the correct one is linked to a Business Offering / Application.

Service Mapping topology updates take longer or may fail completely when CIs are inaccurate or not unique.

Impact on Your Business

Inaccurate CI records can adversely affect the following areas of your business.

Lower MTTR

Data Completeness Incident Reduction

Data Accuracy Increase Operational Visibility

Lifecycle Management

Audit/Compliance

System Integrity Process Automation

Data Accuracy

Better User Experience

User Confidence Most users will notice poor data quality and take away a poor impression of the entire platform without ever seriously reporting the issue.

Getting back a lost first impression is difficult if not impossible

Engagement Questions

Consider the answers to these questions:

What are the major sources for the CI data in your CMDB?

Have you integrated any third-party Discovery Sources into your CMDB? If so, do these integrations use the CMDBTransformUtil / IdentifyAndReconcile REST API?

Do you import data manually? If so, what process do you use? Does your process include the IRE API in the transforms?

Remediation Plays

Summary

Play

Play 1: Analysis

What this play is about

This play helps you find the CIs in your CMDB that have empty (unpopulated) discovery source values.

Required tasks

Use the provided script to find these CIs.

Play 2: (1 of 3 Fix Plays )

Replace missing values in the discovery_source field using the Update Multiple feature

What this play is about

This play tells you how to add the missing discovery source value by using the Update Multiple feature.

Required tasks

Follow the steps to update the records.

Play 3: (2 of 3 Fix Plays)

(Optional) Set the discovery_source field in bulk using a script

What this play is about

Explains the script you can use to bulk update the CIs.

Required tasks

Follow the steps to use the script and test the update.

Play 4: (3 of 3 Fix Plays )

Add IRE to any active Discovery Sources

What this play is about

Explains how you can add scripted calls to the IRE to frequently used Transform Maps.

Required tasks

Follow the steps to create the scripts.

Play 1: Analysis

What this play is about

This play lets you find the CIs in your CMDB that have empty (unpopulated) discovery source values. This play includes a script you can use to find these CIs.

Required tasks

Download the Analysis Script and import as a Fix Script.

Log on to your instance and navigate to System Definition > Fix Scripts. Locate the QueryScript - CIs Processed Via IRE script that you just imported.

Click Run Fix Script and then Proceed . There is no need to use the background option here.

Review results Note here that we r ound the score to the nearest whole number. In this script, the current HSD score is 71.66. Round the score up the 72.

Play 2: Fix Play: Replace the missing values using the Update Multiple feature

What this play is about

This play tells you how to add the missing discovery source (discovery_source) value by using the Update Multiple feature.

Complete the steps in this play on a per-class basis, and repeat this play across all devices in the CMDB.

Roles Required: list_updater, form_admin, asset or itil (to write to cmdb)

Required tasks

Open a single CMDB class in list view

Type cmdb_ci_ class_name .list in the filter navigator search box. Replace class_name with the CI Class you are updating, for example: cmdb_ci_linux_server.list

Click Enter

Click New or navigate to cmdb_ci_ class_name .form

Edit the form layout. Navigate to Banner Context > Configure > Form Layout

Add the discovery_source field to the form layout for new CIs in this class.(You can add this field at the bottom of the page.)

Use the center arrows to move discovery_source from left to right side of slush bucket

Click Save

Return to the List View. Make sure the Discovery Source field is empty for these records (Discovery source = empty)

On one of the column headers, select the context menu and choose Update All

When asked to confirm the update, click OK

A window similar to the following appears. (Your count will be different)

The new form view for this class of devices appears.

The Discovery Source field should be located at the bottom where you added it in step 6 .

Choose the correct value for the records you have chosen in the List View (that is, all of them)

You can see the available choices in the drop down menu as shown in this example

Click Update

Important: Don't change any of the other blank fields. Any changes you make in this form view are applied to ALL records from your previous List View.

In a few moments, the List View reappears, along with a confirmation message. It may take a few moments depending on the number of records being updated.

A window similar to the following appears. (Again, your count will be different.)

Next Steps

Complete the steps in this play as needed for the CIs in each of your classes.

Play 3: Fix Play: (Optional) Set the discovery_source field in bulk

What this play is about

If you have large number of records (approx. 2000+) needing discovery_source values, you can use a script to perform those updates in bulk. Doing this update in bulk helps to avoid performance issues or timeouts.

This play includes a script you can use to complete the bulk update.

The steps used in the play change data in within the target environment. Make sure an approved change window is active. Also, be aware of your company's procedures and be sure to follow them.

Required tasks

Download and import the Bulk CI Update Script.

Navigate to System Definition > Fix Scripts and locate the FixScript - Bulk Update discovery_source script

Review the header comments to understand what is being performed by this script.

Set the following variables at the top of the script:

dryRun – run in test mode or actually update records

updateCount - how large a batch to process at once.

maxIterations – allows for stopping the script after n number of loops

newDiscoSource – the value you wish to set for discovery_source (default: "Manual Entry")

myEncodedQuery – the query that should be used to match records for update

baseClass – if your query requires fields not present on cmdb_ci, change this

Click Save to record your input variables.

Click the Run Fix Script button, and then select the Proceed in Background button.

This option ensures the script does not tie up the system.

Click the Show Progress Workers link which is located after the Script editor.

You will see the list of each time this Fix Script has been launched.

Be sure to sort by Created column in reverse to see the latest one on top. The State column indicates if the script is complete or still running.

Reload the list to check status, and once your latest attempt is showing State of Complete, you can click through into that record using the first field.

Scroll to the bottom of the Message field on your Progress Worker record to view the final output. Much of what is recorded here is not directly related to your script.

You should see something like the following:

*** Script: Found : 2043 records using your query of discovery_source=NULL^nameLIKE64

Successfully updated 2043 records!

[0:00:18.387] Total Time

Return to List View and confirm the updates were correctly made.

The List View you copied the Encoded Query from should reload and be empty now that discovery_source is no longer null or empty.

Play 4: Fix Play: Add IRE to any active Discovery Sources

What this play is about

This play lets you add scripted calls to the IRE to frequently used Transform Maps.

Roles Required: import_admin

This play is designed to manage incoming updates and inserts to the CMDB from external sources, making sure the updates and inserts properly engage the IRE.

The scriptable CMDB API has been available for at least seven major releases. Discovery and Service Mapping already use it.

DataSource Precedence Rules and Reconciliation Rules both require use of the IRE. CMDB accuracy is ensured when all sources align with the CMDB Admin's priorities for varying sources for the same records.

Required tasks

Use this play on each third-party source of data for the CMDB, including manual Imports. Consider retiring any processes that use the legacy Load Data feature. Transition to using a Transform Map with IRE.

Navigate to Transform Maps.

Enter " *computer " into the Target table search box and press enter. The resulting maps should be good candidates for using the IRE.

Note : SCCM 2016 Computer Identity should be patched using the process documented in the Community here.

The following step may need to be performed within the production environment with the help of a ServiceNow customer administrator.

No changes are being performed; This step shows which Import Sets are using Transform Maps targeting the CMDB.

3. Navigate to System Import Sets > Advanced > Transform History

4. Use the Filter to select Transform Map ==> Table Transform Map fields

5. Select Target table - contains - cmdb_ci

6. Click Run. The resulting list should show candidate Transform Maps that should be using the IRE, also known as CMDBTransformUtil.

7. Navigate to System Import Sets > Administration > Transform Maps and open up the map you need to update with IRE.

8. In a separate Browser Tab Navigate to the CMDBTransformUtil documentation. Locate the example, or copy it from here:

// add this code to the onBefore transform map script // Call CMDB API to do Identification and Reconciliation of current row var cmdbUtil = new CMDBTransformUtil(); cmdbUtil.setDataSource('ImportSet'); cmdbUtil.identifyAndReconcile(source, map, log); ignore = true; if (cmdbUtil.hasError()) { var errorMessage = cmdbUtil.getError(); log.error(errorMessage); } else { log.info('IE Output Payload: ' + cmdbUtil.getOutputPayload()); log.info('Imported CI: ' + cmdbUtil.getOutputRecordSysId()); target.get(cmdbUtil.getOutputRecordSysId()); //the above line is not in the web example, but super helpful } 9. On your Transform Map, scroll to the Related Lists, and click New under "Transform Scripts."

10. Change the new record's When drop-down to " onBefore"

11. Paste the contents of the example into the script section, replacing the existing comment. Your window should look like this:

12. Click Submit to save the record

If you open the record to review your work, it will similar to the following:

13. Test the import / transform process end-to-end.

This is to ensure the addition of the IRE / CMDBTransformUtil is working as expected. Also a comment will be present on each row stating that Row Transform was ignored by Script. This is as expected. The work to insert / update records has been handled in the onBefore script, while the Transform itself has ignored our rows of data. The Target Record field should still be useful when checking your work. (Thanks to optional line 16 in our example above)

Play 5: Data Governance Play

What this play is about

This play lists the best practices and processes for CI identification and reconciliation

Required tasks

Periodically review the data in your CMDB and run the Analysis play again to check if there are CIs that were added to the system without going through the IRE.

Congratulations

You have completed this Get Well Playbook.