How to increase the Discovered Items to Configuration Item matching rate - Product Knowledge

How to increase the Discovered Items to Configuration Item matching rate  Product Success Playbook

How to increase the Discovered Items to Configuration Item matching rate

A guide to analyze and remediate low Discovered Items Matching Rates

Table of Contents

Summary

Goal of this Playbook

Audience

Problem Overview

Executive Summary

How this playbook can help you achieve business goals

How this playbook is structured

Problem Analysis

Upstream Causes

Downstream Consequences

Impact on Your Business

Engagement Questions

Remediation Plays

Summary

Play 1: Analyze your DI records

Play 2: Fix for "Field matching" CI matching rules

Play 3: Fix for "Scripted" CI matching rules

Play 4: Fix for "CMDB unprepared"

Data Governance

Summary

Goal of this Playbook

Understand and increase the proportion of Discovered Items (DIs) matched to a Configuration Item (CI).

Important reminders:

DIs get created in a new CMDB table by the Vulnerability Response (VR) application when data is imported from the third party vulnerability scanner. A satisfactory DIs-to-CI matching rate is necessary to get the benefit of risk calculation and assignment feature of VR. Review the principles of a successful VR implementation laid out in this free tutorial " CI Matching - How to do it right ".

Details about this playbook

Author Eric Feron Date 12/15/2021 Addresses HSD # HSD00010218 Applicable ServiceNow Releases Any Time Required Approximately 1 to 8 hours (depending on your environment and number of iterations needed) Audience

Vulnerability Administrator, Vulnerability Analysts, Remediation teams,

ServiceNow Administrator or Discovery Administrator,

Configuration Manager or Configuration Management team.

Problem Overview

As organizations continue to be exposed to fast growing volumes of vulnerabilities, it is critical for their vulnerability profile that vulnerabilities be remedied quickly and efficiently. When, during the import of vulnerabilities from a third-party scanner, DIs are created in the CMDB but not matched to CIs, risk course will not be enriched without having the additional asset context. Once the unmatched issue is fixed, a sudden large quantity of VIs are sent to the remediation teams that will get overwhelmed. It is recommended to reduce the number of unmatched DIs before moving the VR implementation to production and to continue monitoring this metric regularly while in production to keep it at an acceptable level. A low DIs-to-CI matching rate is one of the main and most important obstacles to the good performance of a VR implementation.

Executive Summary

How this playbook can help you achieve business goals

This playbook recognizes the need to reach and maintain an acceptable DIs-to-CI matching rate in order to get best value from a VR implementation. It will help you match un-matched DIs and put in place processes to reduce the recurrence of this issue. This will ensure that the maximum number of vulnerabilities are communicated to the remediation teams in a timely manner for effective treatment. This, is turn, will contribute to improving the vulnerability profile of you organization.

How this playbook is structured

This playbook contains four plays to help you match unmatched DIs.

The 1st play is to show you how to locate the unmatched DIs and the CI Lookup Rules, aka CI matching rules that created them,

The 2th play provides guidance to handle the case when the CMDB is not VR-ready (in sub-production),

The 3rd play provides guidance to rectify the ineffective "Field mapping" matching rules and re-ingest the data (in sub-production and production),

The 4th play provides guidance to rectify the ineffective "Scripted" matching rules, (in sub-production and production),

Lastly, a governance play offers pointers to reduce the future incidence of this issue.

Problem Analysis

Upstream Causes

The drivers of a low DIs-to-CI matching rate are (by decreasing order of business impact):

Unprepared CMDB: - The CMDB is not VR-ready, it has not been aligned to the VR strategy - The network ranges used to discover assets and populate the CMDB are not aligned to the network ranges used by the third party vulnerability scanner - CIs do not exist in the CMDB to match the DIs created by the VR application

Unsuitable CI Lookup Rules - OOTB rules are not fully suited to specific environment and need to be tailored - Rules were created after all data is imported from the scanner (sub-optimal implementation methodology) - Not following the crawl - walk - run recommendation for the creation and enhancement of CI Lookup Rules (sub-optimal implementation methodology)

Downstream Consequences

Data Consequence

Missing relationships

Presence of dormant DIs Operation Consequence

Vulnerabilities remediation takes more time with low matured CMDB due to lack of asset context and assignment rules.

Under-utilized remediation teams

Remediation teams overwhelmed with sudden unplanned work when the issue is fixed

Frustration, lack of confidence in the VR implementation

Impact on Your Business

DIs with no matched CI will negatively impact the effectiveness and efficiency of your Vulnerability Remediation teams, vulnerability teams and audits.

Security MTTR

Delay in vulnerability identification

Slower response to vulnerability remediation

Audit/Compliance

Incomplete DI data

Engagement Questions:

Consider the answers to these questions:

What is the current proportion of unmatched DIs?

Does the VR strategy explicitly address CI Lookup rules?

Did the VR implementation follow the "crawl-walk-run" approach?

Did the CI Lookup Rules strategy use the OOTB rules first?

Is there an established relationship between the VR team and the CMDB team?

Remediation Plays

Summary

The table below lists and summarizes each of the remediation plays in the playbook. Details are included later.

Play Name

Analyze Play

What this play is about

Find unmatched DIs and their CI Lookup Rules Required tasks

Filter lists Fix play for an unprepared CMDB

What this play is about

Ensuring the CMDB is VR-ready Required tasks

Align with CMDB team, fix matching rules or wipe the system clean and start again Fix play for Field matching rules

What this play is about

Rectify the faulty "Field matching" CI Lookup Rules Required tasks

Modify and reapply the CI Lookup Rules Fix play for Scripted rules

What this play is about

Rectify the faulty "Scripted" CI Lookup Rules Required tasks

Modify and reapply the CI Lookup Rules

Data Governance

What this play is about

Limiting future occurrences of the issue Required tasks

Review VR Strategy, engage with the CMDB team.

Play 1 - Analyze your DI records

What this Play is about

Shows you how to view the unmatched DIs and the CI Lookup Rules (aka CI matching rules) that will require rectification.

Required tasks

In the Filter Navigator, search for "Discovered"

Open CMDB > Discovered Items

Group the list by State

Filter to show only State = "Unmatched"

Ensure the list shows the "CI matching rule" column as per example below:

Group the list by CI matching rule as so: This provides a complete view of all the CI matching rules that yield unmatched DIs.

Add a condition to the filter as follows: This will allow you to select either "Field matching" rules or "Scripted" rules for the next plays.

Note:

CI Lookup Rules can also be viewed as so:

In the Filter Navigator, search for "Lookup"

Open CMDB > CI Lookup Rules.

Play 2 - Fix Play for "CMDB unprepared"

What this Play is about

For a sub-production environment, shows you how to address low DIs-to-CI matching rates.

This play assumes that your implementation uses Identification Reconciliation Engine (IRE) , (see article: Reconcile unmatched discovered items ).

Required tasks

Interview the CMDB, Platform and Vulnerability Management teams to identify and itemize the actions already taken (if any) to make the CMDB VR-ready. If the interviews confirm that alignment suitable alignment work has taken place i.e. the CMDB is VR ready, skip to Plays 3 and 4.

If your CMDB is not VR-ready, identify the gaps between the coverage of CMDB discovery network ranges and vulnerability scanner network ranges. This is the primary driver of unmatched DIs.

Plan and prioritize the adjustments needed for the CMDB discovery sources or the VR scanner source data to get aligned. This will require partnership with the Platform and CMDB teams. This effort may require significant time investment. Care should be taken to perform the necessary maintenance (see Data Governance below).

Once the CMDB is VR-ready, reconciliation can continue Follow the directions of Reconcile unmatched discovered items to create a scheduled job to perform the reconciliation.

(Optional) In some cases, or if your implementation does not use IRE, it may be more effective to: Delete all records i.e. purge all VR tables (see the Product Document: Delete all your vulnerable item records and related data in Vulnerability Response . Run VR integration with the scanner again.

CMDB unprepared CMDB ready IRE on see above See below IRE off 1-Turn IRE on

2- See CMDB team as above

3- Purge the DIs and associated CIs (see 5.a and 5.b)

Don't want IRE on

manual fixes only + tool (see JG): CI

Turn IRE on

Purge data

Fix the rules

Dont want IRE on

Notes:

It is recommended to not migrate a sub-production implementation to a production environment if the DIs-to-CI matching rate is unacceptably low. Every effort should be made to ensure that the CMDB is VR-ready, the CI Lookup Rules are effective and very few or no DIs remain unmatched before migration to production. If the VR implementation has been migrated to production before the CMDB was made VR-ready, it is imperative that you contact customer support and seek expert help.

Play 3 - Fix Play for "Field matching" CI matching rules

What this Play is about

For a sub-production or production environment, shows you how to:

Address the "Field matching" CI Lookup Rules that need to be rectified, Reapply the rectified lookup rule, Required tasks

For each CI matching rule identified as showing unmatched DIs, Starting with the one with the largest number of unmatched DIs: After completing play 1, refine the filter as follows:

Starting with the rule showing the largest number of unmatched DIs:

Expand the rule,

Find an unmatched DI that you know should be matched (because you know that the CI exists in the CMDB); The underlying cause is very likely impacting many other (possibly all) unmatched DIs delivered by the same CI matching rule,

Establish why this DI was not matched, i.e. why the rule does not return the right data for the DI. For example, the rule could be too restrictive: - If the FQDN (Fully Qualified Domain Name) rule is restricted to "server1.mycompany.com", when host name = "server1" the rule will return no match for this host, but create an unmatched DI. - In this example, it is recommended to expand the rule to "server1".

Make a note of the name of the CI matching rule to be rectified and the rectification to be implemented,

Go back to the previous list showing all defective CI matching rules,

Click on the "CI matching rule" name,

Click on the "i" radial next to the CI matching rule name,

Click on "Open record",

Rectify the conditions of the rule as needed to ensure the unmatched DI above will get matched to the right CI when the rule is run again. The change in the rule will of course apply to all DIs.

You now need to re-run the CI matching rule that you just rectified so it attempts to match the DIs that it had previously left unmatched. This process is also called "reconciliation":

Go to CI Lookup Rules table and apply the following filters: Choosing "Reapply is true" and running the filter will select the CI matching rules that have not been run, in this case the one that you just rectified.

Click on the "Apply Changes" button; This dialogue box will then appear:

Select "Reapply". This will run the rule you just rectified, and a blue information bar will appear while this is taking place.

Note: there are other ways to execute this reconciliation:

- See KB article: How to reapply CI Lookup Rules

- Via CMDB reconcile as per conditions below, see KB article: Reconcile unmatched Discovered Items

Once the reconciliation is complete, go back to the Discovered Items tables to re-run this filter:

The CI matching rule you just rectified and ran should no longer appear (full success), or show a much smaller number of unmatched DIs (partial success).

Repeat the procedure if needed for this CI matching rule until the number of corresponding unmatched DIs is zero or acceptable.

Repeat the procedure for other "Field matching" CI matching rules.

Note: It is recommended to run the reconciliation after every rectification to a CI matching rule, i.e. not rectify two or more rules and then "Apply changes". This will avoid overloading your system.

Play 4 - Fix Play for "Scripted" CI matching rules

What this Play is about

For a sub-production or production environment, shows you how to:

Address the "Scripted" CI Lookup Rules that need to be rectified, Reapply the rectified lookup rule, Required tasks

For each CI matching rule identified as showing unmatched DIs, Starting with the one with the largest number of unmatched DIs:

After completing play 1, refine the filter as follows:

Follow the exact same steps as for the "Field matching" CI matching rules detailed above. The only difference is that step 2-9 should read: "Rectify the script of the rule as needed to ensure the unmatched DI above will get matched to the right CI when the rule is run again."

Reminder: 100% matched DIs is an ideal to aim for, not a realistic goal to achieve. At maturity level 1, 60-75% is acceptable, at maturity level 2, it is recommended to maintain 80-95% matched DIs. In general, anything above 80% is considered acceptable.

Data Governance

What this Play is about

Now that you have rectified the CI Lookup rules and/or ensured that the CMDB is VR-ready, the play below will help keep the proportion of unmatched DIs to a minimum.

Required tasks

Check play 1 regularly, Establish a schedule of updates and reviews of the VR strategy and implementation with the CMDB team. It is recommended to meet quarterly. Review the principles of a successful VR implementation laid out in this free tutorial " CI Matching - How to do it right ".

Congratulations You have completed this Product Success Playbook.