Handle Duplicate CIs in your CMDB

Get Well Playbook

A guide for how to manage and remediate duplicate CI records

Table of Contents

Summary . 3

Audience . 3

Problem Overview .. 3

Executive Summary . 3

How this playbook can help you achieve your business goals . 3

How this playbook is structured . 3

Problem Analysis 4

Upstream Causes: 4

What are downstream consequences: 4

What is the Impact to Business Outcome: 5

Engagement Questions: 6

Remediation Plays 7

Summary . 7

Analysis Play . 8

Fix Play – Remediate Duplicates using the Wizard . 9

Data Governance . 17

Summary

Goal for the Playbook (Problem or target this play will address):

The goal of this playbook is to help you manage and resolve (remediate) duplicate configuration items (CIs) in your configuration management database (CMDB).

Details about this playbook

Author Johnny Walker Date 06/15/2020 Addresses HSD # HSD0003784, HSD0006973 Applicable ServiceNow Releases All Releases Time required Approximately 1 to 8 hours

Audience

Configuration Manager or Configuration Management team

ServiceNow Admin

Problem Overview

You don't want duplicate CIs in your CMDB. Duplicate CIs can:

Take up valuable space in your CMDB

Create inaccurate data

Erode confidence in the quality of the data

Lead to data loss

Executive Summary

How this playbook can help you achieve your business goals

You need to resolve (remediate) any duplicate CIs. Resolving duplicates ensures that your platform processes are accurate. To help you resolve the duplicates, and ensure their accuracy, use the wizard included in the base system. This wizard merges duplicate records into a single record.

How this playbook is structured

This playbook guides you through three plays.

The first play (an analysis or validation play) shows you how many duplicate CI records you need to remediate.

The second play (a fix play) helps you use the built-in wizard to merge the CI records.

The third play (a governance play) shows you how to use the CMDB dashboard to see the health of the CMDB and make changes, as needed. Problem Analysis

Upstream Causes

Duplicate CIs can be caused by the following:

Discovery patterns fail to gather key data (such as serial numbers, names, and configuration files). These failures may be due to the logic used in the pattern or a credential failure.

CI Identifiers are configured incorrectly.

Data is entered incorrectly (such as typographical errors, failure to check if a CI already exists)

CI naming conditions, or processes for retiring hardware, aren't followed.

Failures occur when importing or integrating data Remediation tasks that you need to address can occur due to a lack of:

CMDB data ownership

Enough people on the Configuration Management team

Diligence for remediating duplicate CI tasks

Knowledge of the CMDB Dashboard

CMDB Dashboard governance

Downstream Consequences

Data Consequence:

Duplicates appear as identical, but separate, records in list view. This can be confusing.

You need to manually remediate any duplicate records (or records that appear empty).

You may update and then compress the wrong CI. If that happens, and the serial number is changed, you lose all history for that CI.

Operation Consequence:

You may need to use special queries to resolve these duplicates. Using special queries adds to your overhead.

Resolving duplicate CIs can be frustrating.

Services you want to use can be adversely affected when processes break down or operate slowly

App Consequence:

Binding and CI selection for Incident Management could be affected.

CI matching for Vulnerability Response (VR) could fail

Change Management or other ServiceNow applications may require correction

Asset Management performance may be adversely affected by broken links to CIs or by having to remediate duplicate CIs.

Impact on Your Business

Resolving duplicate CIs can have an positive impact on the following areas:

Lower MTTR Data validation

CI binding in Event Management

Increase Operational Visibility Data accuracy

Lifecycle management

Audit/Compliance System integrity

Data governance

Process Automation Data accuracy

Integrity of relationships

Better User Experience Platform adoption

Increased confidence in the CMDB Engagement Questions:

Consider the answers to these questions:

Is the owner of the CMDB clearly identified? Who is accountable? Every practice needs to have an owner.

Who can add records? Is it only trained staff?

How large is your Configuration Management team? The CMDB requires maintenance and governance. Who is responsible for those remediation tasks?.

To get the most benefit from your CMDB, someone needs to maintain the data. If there's no one do that, think about what you really need.

Can the Configuration Manager resolve duplicates, which may require deleting, updating, and editing records? If not, who can?

How many hours a week are you allocating to reconciling duplicate CIs? Is this a priority for your company?

What do you do with the duplicate CIs after they are merged to a single CI? Delete them? Archive them? Remediation Plays

Summary

The table below lists and summarizes each of the remediation plays in the playbook. Details are included later.

Play

Analysis

What is this play about

Lets you see if you have any duplicate CIs.

Required tasks

Use list view to see if you have any duplicate CIs.

Fix

What is this play about

Explains how to use the built-in wizard to resolve (remediate) the duplicate CIs.

Required tasks

Use the built-in wizard to update and archive (or remove) inaccurate data

Data Governance

What is this play about

Explains the importance of monitoring the health of your CMDB, and resolving duplicate CIs as soon as you are aware of them.

Required tasks

Use the CMDB Health dashboards to view the duplicate CIs and take action to remediate them.

Then run the CMDB Health Dashboard – Correctness Score Calculation job to update the CMDB.

Analysis Play

What this Play is about

This play helps you identify any duplicate CIs you may have in your CMDB. You can see the duplicate CIs on the Remediate Duplicate Tasks list view.

Note: You can also use a Script Include to identify the duplicate CIs.

Required tasks

Option 1:

Install CMDB and CSDM Data Foundations Dashboard from the ServiceNow App Store

Navigate to the CMDB Data Foundations Dashboard module in the left navigation menu

Select the Data Management Practices tab

Select the Unhandled Duplicate CIs report to view a list of duplicate CIs. If you see a count of zero then this playbook is not applicable to you

Option 2:

To Remediate duplicate tasks, navigate to Configuration > Identification/Reconciliation > De-duplication Tasks

Once the form loads, add the condition of State is Open.

The duplicate CI records appear in list view.

See if you have any duplicate CI records (Remediate Duplicate Tasks).

If you don't have any duplicate CI records, you don't need to do anything else. You have completed the Analysis Play.

If you do have duplicate CI records, you need to reconcile (remediate) them using the built-in wizard. Complete the tasks in the Fix Play. Fix Play

What is Play is about

This play helps you remediate duplicate CI records by using the built-in Duplicate CI Remediator wizard. For more information about this wizard and how duplicates are remediated, see Remediate a de-duplication task .

(Optional) Set system properties

For testing purposes, you can set the system property glide.duplicate_ci_remediator.dry_run to True . When set to True, the changes you select for the wizard won't be written to the CMDB.

If you are working in a production environment, you shouldn't need to set this property but you can if you'd like to. To configure any of the properties related to the wizard, navigate to Configuration > CMDB Properties > Duplicate CI Remediator Properties.

Required tasks

Open one of the Remediate Duplicate Tasks and click Remediate.

The Remediate window appears.

Keep the Use the Duplicate CI RI Remediator (Recommended) option selected.

Click Next to continue. The wizard reviews the CIs and continues with the remediation. The merge process flow begins. Note the following points:

If you make any changes to the CIs now, you must restart the wizard to have the changes take effect.

To restart the wizard, click Cancel and return to step 1 of this play. Review the list of CIs, and select the one you want to use as the Master CI.

(Optional) For help choosing the Master CI, complete these steps. Click the name of the CI to see details. Once these details are displayed, sort the attribute columns in reverse order by using the down arrow next to each column.

When sorted in reverse order, you can see the populated attributes first. These details can help you choose which CI to use as the master.

Decide the remediation method you want to use. You can remediate the CIs manually, or you can use the Master CI.

The table below describes both choices and what to do next based on your choice.

Remediation method

Description

Next steps

Remediate manually

This method lets you specify which attribute values, relationships, and related items from the duplicate CIs to consolidate into the Master CI.

Continue with step 6 and complete all of the remaining steps.

Use Master CI

This method keeps the Master CI attribute values, merges relationships, and merges only the default related items.

This method can save you a lot of time.

Advance to step 12, skipping all the steps in between.

The Merge Attribute Values page identifies any CIs with attribute values that differ from those of the Master CI. To see the value used for the CI, click the Other Value link.

Choose the value you want to keep. Then click Select to return to the Merge Attribute Values page.

Continue selecting values for each attribute, as applicable. When you have finished selecting values, click Next .

The Merge Relationships page appears. The Merge Relationships page appears.

The Merge Relationships page lets you decide if you want to merge all the relationships from the duplicate CIs into the master CI. (To view the relationships, click View all relationships. )

Click the merge option you want. (Merging all relationships is the typical choice, and is most likely the one you want.) Note the following points:

With either choice, all orphaned or duplicate relationships are deleted.

Once the relationships are merged, it can't be reversed.

Relationships created by Discovery or Service Mapping are recreated the next time the CMDB is scanned. Scroll to the Merge Related Items page. The Merge Related Items are task records that reference the merged CIs. (To view all the related tasks, click View all related items .)

Use the center arrows to move task_ci from the Available list to the Selected list. The task_ci is a supporting record for the incident listed in the example. above. The main CI on a task is automatically copied to the task_ci for that record. You need both.

(The CMDB Health Result item is a remediate_duplicate_task used at the beginning of this play. You can disregard it.)

Click Next. The Duplicate CI Actions page appears.

On the Duplicate CI Actions page, choose what you want to do with the merged CI records.

You can have the merged CI records deleted or (if you are using data archiving ) you can set custom values in certain fields. Note the following points:

With data archiving, you can set a specific value in one of the fields. Data matching this value can be automatically archived based on an archive rule you create. For more information about archive rules, see Create an archive rule .

Delete any incomplete or obsolete records.

Always follow your company's policy on data retention. Click Next . The Review and Confirm page appears.

Review the page to make sure you haven't overlooked anything. Then click Remediate. The following completion message appears:

"Orphan and duplicate relationships will be deleted.[...]"

This is what you want. You have successfully remediated the duplicate CIs.

Data Governance Play

What this Play is about

This play explains the importance of monitoring the health of your CMDB, and resolving duplicate CIs as soon as you are aware of them. The options on the CMDB dashboard can help you do that.

Required tasks

Navigate to Configuration > CMDB Dashboard > CMDB View.

Scroll to the Duplicate CIs widget to view the health of your CMDB.

Use this chart to see the number of duplicate CIs in each class. Click the name of the class you want to view.

In the List View, click a task record to view the details. Note: If you need to remediate the record, follow the steps in the Fix Play. Any changes you make will appear the next time you run the CMDB Health Dashboard – Correctness Score Calculation job.

IMPORTANT: The scores (including the ones in the Duplicate CIs widget) are immediately updated. Therefore, run this job outside of the normal production hours.

Click Execute Now .

Congratulations

You have completed this Playbook.