How does De-duplication Template work ?Summary<!-- /*NS Branding Styles*/ --> .ns-kb-css-body-editor-container { p { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } span { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } h2 { font-size: 24pt; font-family: Lato; color: var(--now-color--text-primary, black); } h3 { font-size: 18pt; font-family: Lato; color: var(--now-color--text-primary, black); } h4 { font-size: 14pt; font-family: Lato; color: var(--now-color--text-primary, black); } a { font-size: 12pt; font-family: Lato; color: var(--now-color--link-primary, #00718F); } a:hover { font-size: 12pt; color: var(--now-color--link-primary, #024F69); } a:target { font-size: 12pt; color: var(--now-color--link-primary, #032D42); } a:visited { font-size: 12pt; color: var(--now-color--link-primary, #00718f); } ul { font-size: 12pt; font-family: Lato; } li { font-size: 12pt; font-family: Lato; } img { display: ; max-width: ; width: ; height: ; } } In the ServiceNow Vancouver release, we released a new feature called as de-duplication template. This KB explains the working of the template. Release<!-- /*NS Branding Styles*/ --> .ns-kb-css-body-editor-container { p { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } span { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } h2 { font-size: 24pt; font-family: Lato; color: var(--now-color--text-primary, black); } h3 { font-size: 18pt; font-family: Lato; color: var(--now-color--text-primary, black); } h4 { font-size: 14pt; font-family: Lato; color: var(--now-color--text-primary, black); } a { font-size: 12pt; font-family: Lato; color: var(--now-color--link-primary, #00718F); } a:hover { font-size: 12pt; color: var(--now-color--link-primary, #024F69); } a:target { font-size: 12pt; color: var(--now-color--link-primary, #032D42); } a:visited { font-size: 12pt; color: var(--now-color--link-primary, #00718f); } ul { font-size: 12pt; font-family: Lato; } li { font-size: 12pt; font-family: Lato; } img { display: ; max-width: ; width: ; height: ; } } Instance with CMDB Workspace Instructions<!-- /*NS Branding Styles*/ --> .ns-kb-css-body-editor-container { p { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } span { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } h2 { font-size: 24pt; font-family: Lato; color: var(--now-color--text-primary, black); } h3 { font-size: 18pt; font-family: Lato; color: var(--now-color--text-primary, black); } h4 { font-size: 14pt; font-family: Lato; color: var(--now-color--text-primary, black); } a { font-size: 12pt; font-family: Lato; color: var(--now-color--link-primary, #00718F); } a:hover { font-size: 12pt; color: var(--now-color--link-primary, #024F69); } a:target { font-size: 12pt; color: var(--now-color--link-primary, #032D42); } a:visited { font-size: 12pt; color: var(--now-color--link-primary, #00718f); } ul { font-size: 12pt; font-family: Lato; } li { font-size: 12pt; font-family: Lato; } img { display: ; max-width: ; width: ; height: ; } } Table of Contents Step 1: Define and Publish the PolicyStep 2: Task Selection MechanismStep 3: Executing the PolicyTroubleshooting and Performance considerations Step 1: Define and Publish the Policy The first is to create a De-duplication Template with valid criteria and attribute mapping. Kindly follow the steps mentioned in "Create a de-duplication template" document The default status of the template is draft. Once everything is updated, kindly publish the template. Note - Make sure you select "Automatically select all de-duplication tasks that match the selected class" checkbox while defining the policy to automatically select the de-dup task for processing Step 2: Task Selection Mechanism OOTB, in the latest release there is a new field called "Attributes JSON Blob" added to the "reconcile_duplicate_task" table. This field stores the JSON object of the duplicate task created by the IREThe scheduled Job "De-duplication: Populate Duplicate Task Data" runs daily and uses this JSON object to create a new record in reconcile_duplicate_task_data table that contains the attributes and the keys 3. Once the data is populated in the reconcile_duplicate_task_data table, the same job deletes the JSON data from the reconcile_duplicate_task table. Step 3: Executing the Policy When the policy is executed, the policy selects the tasks from the reconcile_duplicate_task_data table and processes the tasks as per the defined configuration. Note - The policy does not consider all the tasks from the reconcile_duplicate_task table. It only considers those tasks that are processed by the "De-duplication: Populate Duplicate Task Data" scheduled Job job and have an entry in the reconcile_duplicate_task_data table. Troubleshooting and Performance considerations You will see these execute in Scheduler Worker threads, with the name "ASYNC: Script Job", where the Job Context contains the Script:sn_cmdb.DuplicateTemplate.remediateTaskRun(<sys_id of reconcile_duplicate_template_task_run record>, g_schedule_record.sys_id); The stack trace while one of these runs will look roughly like: ...com.glide.db.DBQuery.execute(DBQuery.java:3653)com.snc.cmdb.duplicate_merge.RelatedItemMerger.executeQuery(RelatedItemMerger.java:1065)com.snc.cmdb.duplicate_merge.RelatedItemMerger.getRelatedItemCount(RelatedItemMerger.java:355)com.snc.cmdb.duplicate_merge.TaskDataManager.fetchRelatedItemCount(TaskDataManager.java:129)com.snc.cmdb.duplicate_merge.CmdbCiTaskDataManager.(CmdbCiTaskDataManager.java:64)com.snc.cmdb.duplicate_merge.DuplicateTaskData.getDataManager(DuplicateTaskData.java:122)com.snc.cmdb.duplicate_merge.DuplicateTaskData.(DuplicateTaskData.java:102)com.snc.cmdb.duplicate_merge.DuplicateTaskData.(DuplicateTaskData.java:84)com.snc.cmdb.duplicate_merge.template.TemplateExecutor.init(TemplateExecutor.java:63)com.snc.cmdb.duplicate_merge.template.TemplateExecutor.(TemplateExecutor.java:50)com.snc.cmdb.duplicate_merge.template.TemplateExecutor.(TemplateExecutor.java:43)com.snc.cmdb.duplicate_merge.template.TemplateTaskRunExecutor.execute(TemplateTaskRunExecutor.java:40)com.snc.cmdb.duplicate_merge.template.DuplicateTemplateScriptableApi.jsStaticFunction_remediateTaskRun(DuplicateTemplateScriptableApi.java:172)... This is a bulk feature, so lots of these jobs could be put in the sys_trigger table queue at the same time, and potentially cause a performance issue. Lots of the limited number of worker threads in-effect blocked while these jobs run concurrently. Other jobs may be delayed.Large or slow CMDB queries, returning large result sets, using potentially large amounts of memory on the database servers. There are 2 properties that will limit the concurrency of these jobs, so fewer will be running at the same time. If you do have performance issues, they can be mitigated by lowering these default values. The jobs will all still run, but spread the work over more time, limiting the impact: glide.dedupe_template.max_template_concurrency=5glide.dedupe_template.max_task_concurrency=10 These will not be present in the sys_properties table out-of-box, and will need creating manually. Related Links<!-- /*NS Branding Styles*/ --> .ns-kb-css-body-editor-container { p { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } span { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } h2 { font-size: 24pt; font-family: Lato; color: var(--now-color--text-primary, black); } h3 { font-size: 18pt; font-family: Lato; color: var(--now-color--text-primary, black); } h4 { font-size: 14pt; font-family: Lato; color: var(--now-color--text-primary, black); } a { font-size: 12pt; font-family: Lato; color: var(--now-color--link-primary, #00718F); } a:hover { font-size: 12pt; color: var(--now-color--link-primary, #024F69); } a:target { font-size: 12pt; color: var(--now-color--link-primary, #032D42); } a:visited { font-size: 12pt; color: var(--now-color--link-primary, #00718f); } ul { font-size: 12pt; font-family: Lato; } li { font-size: 12pt; font-family: Lato; } img { display: ; max-width: ; width: ; height: ; } } This feature is available in CMDB Workspace Plugin 5.0.4 and beyond.