CMDB Health - Duplicate Metric algorithmDescriptionNot all CIs are checked by the Duplicate Metric when the "CMDB Health Dashboard - Correctness Score Calculation" job runs. When CMDB Health - Correctness job runs, the Duplicate Metric only checks CIs that have Identification Rules defined, and also meet one of below conditions: One of the duplicated CIs has empty Discovery Source attributeThere is an active de-duplicate task (reconcile_duplicate_task) for the duplicated CI To fix all the left-over CIs, please run discovery, or any custom scripts / import engine that calls IRE (Identification and Reconciliation Engine). NOTE: The below Examples are out of date, and applies to instance release prior to New York. In New York, we have introduced a new field: duplicate_of. However, the logic is still very similar to below. The bottom line being that if both CIs have Discovery Source value not empty they will not be detected, regardless of the value in duplicate_of attribute. Why is this the case that the discovery source must be empty for one of the CIs? Configuration Items (CIs) are expected to be inserted via calls to the identification engineDiscovery source is a field which is meant to be used by the identification engine, for example to apply reconciliation rulesWhen a call is made to the identification engine, the IRE will create the duplicate tasks and flag duplicate CIsAs this is already done at insertion, there is no need to check such CIs again as a starting point if such CI has the discovery_source field populatedThe Duplicate Metric job would take considerable time and resources if all records were checked as a starting point as the CMDB grows to millions of records Logic Is this class independent (This is based on the identification rules for the class, review the CI identifier to determine if the class is independent)? Yes: ContinueNo: End logic Query CIs that meet ALL following conditions: ((Discovery_Source = Unknown AND (SYS_UPDATED_ON 1 day ago)) OR (Discovery_Source is Empty))Duplicate_Of is EmptyMeets the conditions from the inclusion rules, cmdb_ie_active_config, for that class Iterate over the list of CIs returned in the query For each CI call the identification engineThe identification engine will determine if duplicate and if so create a de-duplication task Example There are two CIs, having same name and serial number, when duplicate metric runs, the result is based on discovery_source and reconcile_duplicate_task: Case 1: CI01 Discovery Source: ServiceNowCI02 Discovery Source: ServiceNow Result: not detected, unless there is a deduplicate task (reconcile_duplicate_task) Case 2: CI01 Discovery Source: ServiceNowCI02 Discovery Source: Duplicate Result: not detected, unless there is a deduplicate task (reconcile_duplicate_task) Case 3: CI01 Discovery Source: SomeSource (discovery source that doesn't exist in sys_choice table)CI02 Discovery Source: Duplicate Result: not detected, unless there is a deduplicate task (reconcile_duplicate_task) Case 4: CI01 Discovery Source: ServiceNowCI02 Discovery Source: (empty) Result: detected, empty changed to Duplicate, deduplicate task created Case 5: CI01 Discovery Source: (empty)CI02 Discovery Source: (empty) Result: detected, one changed to Duplicate, the other changed to Unknown, deduplicate task createdResolutionCI not marked as duplicate by the daily duplicate processor job Does the CI belong to a class that is independent? Yes: ContinueNo: The CI will not be reviewed by the processor and not marked duplicate Does the CI meet ALL the following conditions? ((Discovery_Source = Unknown AND (SYS_UPDATED_ON 1 day ago)) OR (Discovery_Source is Empty))Duplicate_Of is EmptyMeets the conditions from the inclusion rules, cmdb_ie_active_config, for that class Yes: ContinueNo: The CI will not be reviewed by the processor and not marked duplicate Does the CI actually have a duplicate? Yes: The IRE will determine which one is the duplicate and create a de-duplication taskNo: No task created In order to find duplicates, the Duplicate Processor calls the Identification and Reconciliation Engine (IRE) using 'Manual Entry' as the data source in the IRE call. If the out of box 'Manual Entry' option is deleted, when processing duplicate metric, the IRE will throw error: identification_engine : INVALID_INPUT_DATA In payload invalid data source [Manual Entry] exist. You need to provide a valid choice value from field [discovery_source] in table [cmdb_ci]. Thus, please make sure the out of the box Discovery Source 'Manual Entry' is not changed or deleted. Also refer to KB0712338 which explains the workflow of Duplicate Metric when health job runs.Additional InformationThis can also happen when there are orphan records: Check instance system "log > all > source" CMDBHealth, it might give record table name and last process id which is the sys_id of the record. For this record, on de-duplicate tasks table, find the related task, and check if you can open all the CIs under duplicate audit results.On cmdb list view, search for class is <table name>, compare the total record number with the number for <table name> list view. If different, then there may be orphaned records.Enable the system property glide.cmdb.logger.source.identification_engine, of type string, with value = info,warn,error,debug,debugVerbose. The instance node log will provide more information.Debug property: glide.cmdb.logger.source.CMDBHealth