Percent of Active Hardware and VM Instance CIs processed via IRE - Product Knowledge

Percent of Active Hardware and VM Instance CIs processed via IRE  Get Well Playbook

Percent of Active Hardware and VM Instance CIs

processed via IRE

A guide for how to integrate third party discovery sources correctly

Table of Contents

Summary

Audience

Before you begin

Problem Overview

Executive Summary

How this playbook can help you achieve your business goals

How this playbook is structured

Problem Analysis

Upstream Causes

Downstream Consequences

Impact on Your Business

Engagement Questions

Remediation Plays

Summary

Play 1: Analysis

Play 2: (Fix Play) Update existing Transform Maps to use CMDBTransformUtil

Play 3: (Fix Play) Modify Web Service Integrations to use the IdentifyAndReconcile API

Data Governance

Summary

Goal of this Playbook

The goal of this playbook is to help you identify third party sources of discovery data and update them to make use of the Identification and Reconciliation Engine (IRE) . The IRE is designed to maintain the integrity and accuracy of your CMDB by avoiding duplicates while also ensuring that the correct records are updated or inserted. The goal should be a CMDB with fresh, accurate and complete data.

Details about this playbook

Author John (Johnny) Walker Date 08/18/2020 Addresses HSD # HSD0006970 Applicable ServiceNow Releases Orlando, Paris or later Time required Approximately 4 to 20 hours depending on how many integrations you have

Audience

ServiceNow Admin

Configuration Manager or Configuration Management Team

ServiceNow Integration Developers

Problem Overview

In today's world, there is great desire to make use of as many sources of data to populate the CMDB as possible. When incoming discovery data passes through the IRE, it interacts with Datasource Precedence Rules, Reconciliation Rules, CI Identifiers and other Metadata Relationships to ensure that correct records are placed completely within the CMDB. These rules and records work together to provide fine-tuned control over multiple sources of discovery data.

When third party sources (sources other than ServiceNow Discovery and Service Mapping) are allowed to write directly to the CMDB tables, or when transform maps are configured to work directly against CMDB tables, the resulting records and updates can be quite confusing and erratic. There is far greater risk that duplicates will be created due to failure to identify existing matches. When key attributes are missing, incomplete records can be generated when they instead should be skipped until sufficient information is available for identification.

Legacy integrations with third party sources for discovery might attempt to coalesce incoming data records to the CMDB using fields like serial number. That may work most of the time, but those serial numbers exist within at least two different places, making coalesce difficult or cumbersome. Additionally, a coalescing transform map will fail to make use of the provided invalid serial numbers feature.

Having multiple sources of discovery data is a good idea in theory, but in practice it is usually true that one source should be considered more reliable than another for a given class of devices. Conversely, the same source might be less reliable for another class of device. What happens when source A and source B disagree about a particular attribute such as the OS Version? Without the IRE and its rules, you will experience "flapping" as each integration toggles the same fields to differing values, again and again. Perhaps if source A lacks any data for a record over a 30 day span, then source B should be considered accurate enough to perform an update. Flexibility like this is built into the platform through the IRE feature.

If you are not using the IRE, you are missing out on a large amount of value in terms of automated governance and granular control over your discovery sources. The result will always be questionable data within the CMDB and a poor user experience.

Executive Summary

How this playbook can help you achieve your business goals

This playbook will provide step by step instructions to enable you to leverage the IRE to maintain the integrity of the Configuration Management Database (CMDB) while also making the best possible use of all of your sources for discovery data.

How this playbook is structured

This playbook guides you through through five plays.

Play 1 (an analysis play) helps you find the CIs in your CMDB that have empty (unpopulated) discovery source values.

Play 2 (a fix play) shows you how to add scripted calls to the IRE from frequently used Transform Maps.

Play 3 (a fix play) shows you how to send third party integrations through the IRE using the provided REST endpoint

Play 4 (a Data Governance play) lists the guidelines and processes for CI identification and reconciliation. Use these guidelines to maintain the health of the CMDB.

Problem Analysis

Upstream Causes

The scriptable IRE or CMDBTransformUtil wasn’t available when you first started using the ServiceNow platform, and you haven’t been able to overhaul your legacy integrations.

Some third-party integration vendors have not yet adopted the IRE, as is now recommended.

Home grown integrations were built using Data Sources and Transform Maps but use of the CMDBTransformUtil was overlooked.

You may not be aware that your manual processes and data imports can be automated, as well as processed through the IRE.

Downstream Consequences

Data Consequence

If the wrong CI is chosen to match and update, someone else's CI just vanished, overwritten by the data from another Config Item

Failing to match existing CIs can create duplicates

Scheduled Imports can repeat these duplications weekly or even daily

Operational Consequence

Bad data erodes trust in the accuracy of the CMDB

Processes that rely on the accuracy of the CI data in the CMDB are compromised

You might need to create special queries to filter data around duplicates, this additional effort will slow all daily operations

Remediating duplicate CIs takes valuable time away from other tasks

Application Consequence

Most ServiceNow applications depend on accuracy within the CMDB

Asset Management suffers due to duplicate assets or incorrect linkages between CI and Asset

Event Management Alerts can't bind correctly to Cis if there are duplicate records for the event node

Change Requests are only reliable during an audit if the CMDB data is reliable

Change Collision Avoidance will fail when two changes are linked to opposite duplicates

Change automation / groupings can fail when server information is duplicated within the CMDB

Service Mapping topology updates take longer or may fail completely when CIs are inaccurate or not unique.

List views will be cluttered with records that are incomplete, inaccurate or duplicated

Remediate Duplicate CI tasks will pile up, demanding time and attention Impact on Your Business

Inaccurate CI records can adversely affect the following areas of your business.

Lower MTTR

When the data within your CMDB is accurate and complete, resolving incidents will be more efficient because the records will be much easier to locate

Event Management will more accurately bind alerts to config items when they are correct, complete and unique Incident Reduction

When all data within the CMDB is accurate, there is far less risk of selecting the wrong configuration items for changes Increase Operational Visibility

The dashboards for managing the CMDB will show the most accurate and up to date information when all sources of data are in agreement

Audit/Compliance

Reporting and auditing will be reliable when all sources of data are correctly integrated to the CMDB Process Automation

All digital workflows rely upon the data within the CMDB to correctly and accurately route tickets, approvals and notifications

Better User Experience

Most users will notice poor data quality and take away a poor impression of the entire platform without ever seriously reporting the issue

Once it is known that that integrity of the data within your CMDB is of paramount importance, users will enjoy knowing they can rely upon the information they are using across all of the platform applications Engagement Questions

Consider the answers to these questions:

What are the external sources for the CI data in your CMDB?

Do your integrations use the CMDBTransformUtil / IdentifyAndReconcile REST API?

Is it known which sources have the most accurate information, and for which classes of config items?

Do you import data manually? If so, what process do you use? Does your process include the IRE API in the transforms?

Once an asset is entered as on-order, or placed into service, what process is in place to ensure that eventually that record is updated by Discovery or a third party source? Remediation Plays

Summary

Play

Play 1: Analysis

What this play is about

This play helps you find the CIs in your CMDB that have empty (unpopulated) discovery source values.

Required tasks

Use the provided script to find these CIs.

Play 2: Add IRE to transform map integrations using CMDBTransformUtil

What this play is about

Explains how you can easily add scripted calls to the IRE to your Transform Maps.

Required tasks

Follow the steps to update your maps.

Play 3: Add IRE to WebService integrations using the Identification and Reconciliation API

What this play is about

Explains how WebService integrations can make use of the provided REST service for IRE.

Required tasks

Learn from the provided guidance and prepare to update your integrations.

Play 4: Governance and guidance for ensuring that new integrations utilize the IRE

What this play is about

Guidance on how to plan integrations for third party discovery data.

Required tasks

Read, discuss, learn and govern.

Play 1: Analysis

What this play is about

This play lets you find the CIs in your CMDB that have not been processed by the Identification and Reconciliation Engine (IRE).

Required tasks

Download the Update Set and install it in your instance. This update includes a Database View and a sample Report.

Navigate to Configuration > CMDB Dashboard > Config items not through IRE to load a report we have provided for this playbook.

If you do not see the option, you may need to refresh your browser tab to pick-up the new menu item.

Example report:

Notice in this example report output that there are quite a few records listed here, sorted by Location. You can change the sorting on the report to other useful fields such as Managed by, Class or Created by. For more information on any of these groups of records, just click the section of the pie chart to open a list view of those records.

This report is backed by a Database View , which means the records in that view will be read-only. For our purposes this is OK because we are concerned in the following plays with updating the integrations / processes that created these records.

Additionally, a module has been added to your navigation menu under Configuration > CMDB Dashboard > Config Items not through IRE which can be used to navigate to the database view directly with a filter condition of just the Hardware and VM Instance configuration items. Remember that the database view is a join of two of more tables and is read-only. We simply use this view to help us track our progress in adjusting our third party discovery sources to process records through the identification and reconciliation engine. We are not trying to edit the configuration items while inside this view.

With the report open, consider the following:

Is there a particular class of device that seems to be present here in large numbers? Why would that be?

Are most of the devices listed here in a particular location(s)? What is the source of these records?

When sorted on the Created by field, does it become more clear where records are entering the CDMB without properly passing through Identification & Reconciliation?

When sorted on the Managed by (or by group) field, is it clear which teams might be interested in the greater accuracy provided by using the IRE?

Make note of the largest groupings of these records, and after determining what integration or process created them, continue on to the appropriate play below to incorporate use of the IRE. Play 2: Add IRE to transform map integrations using CMDBTransformUtil

What this play is about

This play lets you add scripted calls to the IRE to frequently used Transform Maps.

Roles Required: import_admin

Required tasks

Use this play on each third-party source of data for the CMDB, including manual Imports. Consider retiring any processes that use the legacy Load Data feature. Transition to using a Transform Map with IRE.

Navigate to System Definition > Fix Scripts and search for "Transforms lacking IRE". (This was installed as part of the Update Set listed in the previous analysis play.)

Click on Run Fix Script to get a list of candidate Transform Maps for update.

Example output: *** Script: Transform Map: "Computer" appears to NOT be using the CMDBTransformUtil. *** Script: URL: https://SOMEINSTANCE.service-now.com/sys_transform_map.do?sys_id=4e3548e2c0a80094002e0c1550d11a26 *** Script: This map has no recent histories in this particular instance. Is it used in other environments? *** Script: **** **** **** Notice here that no recent histories were located in our current instance, but if you are aware that a Transform Map is used within another environment, such as Production, you should continue with this play to update this or any other map listed here. 3. Navigate to System Import Sets > Administration > Transform Maps and open up the map you need to update with IRE.

4. In a separate Browser Tab Navigate to the CMDBTransformUtil documentation. Locate the example, or copy it from here:

// add this code to the onBefore transform map script // Call CMDB API to do Identification and Reconciliation of current row var cmdbUtil = new CMDBTransformUtil(); cmdbUtil.setDataSource('ImportSet'); cmdbUtil.identifyAndReconcile(source, map, log); ignore = true; if (cmdbUtil.hasError()) { var errorMessage = cmdbUtil.getError(); log.error(errorMessage); } else { log.info('IE Output Payload: ' + cmdbUtil.getOutputPayload()); log.info('Imported CI: ' + cmdbUtil.getOutputRecordSysId()); target.get(cmdbUtil.getOutputRecordSysId()); //the above line is not in the web example, but super helpful } 5. On your Transform Map, scroll to the Related Lists (which is at the bottom of the form view), and click New under “Transform Scripts.”

6. Change the new record’s When drop-down to “ onBefore”

7. Paste the contents of the example into the script section, replacing the existing comment. Your window should look like this:

8. Click Submit to save the record

If you open the record to review your work, it will similar to the following:

9. Test the import / transform process end-to-end.

This is to ensure the addition of the CMDBTransformUtil is working as expected. Also, a comment will now be present on each imported row stating that 'Row Transform was ignored by Script.' This is as expected. The work to insert / update records has been handled in the onBefore script, while the Transform itself has ignored our rows of data. The Target Record field should still be useful when checking your work. (Thanks to optional line 16 in our example above)

Even if your initial tests look successful, it is often important to allow your testers to review the resulting data for a few days to ensure that any change to your system is working as intended with no other unexpected consequence.

Play 3: Add IRE to WebService integrations using the Identification and Reconciliation API

What this play is about

This play is concerned with enabling integrations to use the IRE through a platform provided REST api.

Roles Required: admin

It may be that the source of incoming records and updates to your CMDB is an integration using the REST Table API . While this API is available on most tables, including the CMDB, it is better practice to make use of the provided IdentificationEngineScriptableApi . This scriptable API is a WebServices version of the IRE and it's already part of your CMDB. You can use it right now.

Required tasks

If you are unsure if any integrations are writing to your CMDB through table API calls, you can easily check the transaction logs.

Navigate to System Logs > Transactions.

Open the filter and add the following conditions, and click 'Run'

Field Operator Value Created on Today URL contains /api/now/table/ URL contains cmdb_ci Type is REST Evaluate the resulting list view. If there are no records listed here, try changing the Created on range to "This month". If you still don't see any records, perhaps you need to check in your Production instance where the integrations are known to be active. It may be that you don't have any transactions in your lower environment unless you coordinate with the remote system's administrator to have them enabled for testing.

Example:

Notice in this example which ID created the transaction. Here it says 'admin', but most likely in your environment you will see the name of an integration user account. The username here should give you some clue as to what system is creating records. It might even match the user id that created many of the records from Play 1, above.

Here you need to familiarize yourself with use of the WebService for IRE. To do that, let's use the REST API Explorer.

Navigate to System Web Services > REST > REST API Explorer

In the top left, use the API Name drop down to select "Identification and Reconciliation"

On the right side, enter a Discovery Source value in the indicated box for the query parameter: sysparm_data_source.

Example: Still on the right side, scroll down to the 'Request Body' section, and click on 'Raw'. This will allow you to paste an example payload for Identification and Reconciliation.

Note: You can generate a payload manually through the identification simulation tool.

Paste the payload into the blank under the Request Body, in the "Raw" input box.

Example:

Click the 'Send' button. A warning will appear to confirm that you are about to modify data. This is expected, since we are about to send information to the IRE to be Identified and Reconciled into the CMDB. You should probably be attempting this step in a non-production environment.

Example:

Click 'OK' to send / test your payload item. The resulting page will show you the output from this web service. This process of generating / testing Web Service calls should be helpful to you and your development team as you uplift your legacy integrations to make use of the provided IRE features through this Web Service.

Note: Web Service development should take place utilizing your own company's demand and/or release management processes. Care should also be taken to check the ServiceNow store for an existing integration that already exists for the product you wish to integrate. Play 4: Data Governance Play

What this play is about

This play lists the best practices and processes for CI identification and reconciliation

Required tasks

Periodically review the data in your CMDB and run the Analysis play again to check if there are CIs that were added to the system without going through the IRE.

Congratulations

You have completed this Get Well Playbook.