Threat Intelligence Security Center Entities De-duplication logicSummary<!-- /*NS Branding Styles*/ --> .ns-kb-css-body-editor-container { p { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } span { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } h2 { font-size: 24pt; font-family: Lato; color: var(--now-color--text-primary, black); } h3 { font-size: 18pt; font-family: Lato; color: var(--now-color--text-primary, black); } h4 { font-size: 14pt; font-family: Lato; color: var(--now-color--text-primary, black); } a { font-size: 12pt; font-family: Lato; color: var(--now-color--link-primary, #00718F); } a:hover { font-size: 12pt; color: var(--now-color--link-primary, #024F69); } a:target { font-size: 12pt; color: var(--now-color--link-primary, #032D42); } a:visited { font-size: 12pt; color: var(--now-color--link-primary, #00718f); } ul { font-size: 12pt; font-family: Lato; } li { font-size: 12pt; font-family: Lato; } img { display: ; max-width: ; width: ; height: ; } } Steps For Checking Duplicates of Source Records Types of Field Level De-Duplication Compare even if fields are blank: Fields of two source records are compared even though they are blank while checking if two source records are duplicate of each other. Eg: If Description of sourceRecord1 is blank and Description of sourceRecord2 is not blank. On the basis of Description Field sourceRecord1 is not duplicate of sourceRecord2 Compare if fields are not blank: Fields of two source records are compared only if either of the source records field value is not blank while checking if two source records are duplicate of each other. In case the field value of one of the source record is blank and the field value of other source record is not blank , the source record with blank field value is considered as duplicate of source record with non-blank field value. (When De-duplication is done On the basis of this field) Eg: If Description of sourceRecord1 is blank and Description of sourceRecord2 is not blank. On the basis of Description Field sourceRecord1 is considered as duplicate of sourceRecord2. If both the source records have some value populated in Description , source records will be considered duplicate of one another only if value from both the source records is one and the same. Compare if field values are subset: If field values of one source record are subset of field values of other source record , the source record whose field values are subset is considered as duplicate of other source record(superset of field values). Eg: If sourceRecord1 has tags t1,t2,t3,t4 and sourceRecord2 has tags t2, t4. Since t2,t4 is subset of t1,t2,t3,t4 , SourceRecord2 is considered as Duplicate of SourceRecord1 when duplication nis checked on basis of tags Common De-Duplication Logic For all Entities De-Duplication based on Relationships and Granular Markings is common step while trying to identify duplicate source records for all the entities. If two source records are identified as duplicates based on field values , then the source records are validated for duplication based on Relationships and Granular Markings Duplication Based on Relationships: When duplication is checked on the basis of relationships , two Source records are considered duplicate of each other only if one of the source records contributes same relationships/ subset of relationships as other source to the aggregated record. The source record contributing subset / same relationships as other source is considered duplicate of the other source contributing super set of relationships Duplication Based on Granular Markings: When duplication is checked on the basis of granular markings , two Source records are considered duplicate of each other only if one of the source records contributes same granular markings / subset of granular markings as other source to the aggregated record. The source record contributing subset / same granular markings as other source record is considered duplicate of the other source contributing super set of granular markings Entity Field Wise Deduplication Logic Observable source De-duplication Type of Deduplication Fields Compare even if fields are blank Value, Type, Source Reported Score Compare if fields are not blank Description, Reputation, Additional Context, Notes, First Observed, Last Observed, First Seen, Last Seen, Threat Level, Threat Severity, Confidence and observable type specific attributes Compare if field values are subset Usage Categories, Tags, Taxonomies, Attack Phases, Authors Indicator Source De-duplication Type of Deduplication Fields Compare even if fields are blank Name, Pattern , Pattern Type Compare if fields are not blank Description, Pattern Version, Additional Context, Notes, Confidence, First Observed, Last Observed, First Seen, Last Seen, Threat Level, Threat Severity, First Detected, IOC Classification Compare if field values are subset Usage Categories, Tags, Taxonomies, Attack Phases, Platforms, Indicator Types, Authors Object Source De-duplication (Common fields for all objects) Type of Deduplication Fields Compare even if fields are blank Spec Version, Confidence Compare if fields are not blank Extensions, Notes, Additional Context, Threat Level, Threat Severity Compare if field values are subset Tags, Taxonomies Vulnerability Source De-duplication Type of Deduplication Fields Compare even if fields are blank Name, Severity Compare if fields are not blank Description, CVSS2.0, CVSS3.0, Exploitation Status Compare if field values are subset Affected Software Threat Event Source De-duplucation Type of Deduplication Fields Compare even if fields are blank Name Compare if fields are not blank Description Compare if field values are subset - Identity Source De-duplication Type of Deduplication Fields Compare even if fields are blank Name, Identity Class Compare if fields are not blank Description, Contact Information Compare if field values are subset Sectors, Roles Threat Opinion Source De-duplication Type of Deduplication Fields Compare even if fields are blank Opinion Compare if fields are not blank Explanation Compare if field values are subset Authors Location Source De-duplication Type of Deduplication Fields Compare even if fields are blank Name Compare if fields are not blank Description, Latitude, Longitude, Precision, Region, Country, Administrative Area, City, Street Address, Postal Code Compare if field values are subset - Attack Pattern Source De-duplication Type of Deduplication Fields Compare even if fields are blank Name Compare if fields are not blank Description Compare if field values are subset Aliases, Attack Phases, Permissions Required Course of Action Source De-duplication Type of Deduplication Fields Compare even if fields are blank Name Compare if fields are not blank Description, Action Compare if field values are subset External ID Intrusion Set Source De-duplication Type of Deduplication Fields Compare even if fields are blank Name Compare if fields are not blank Description, Resource Level, Primary Motivation, First Seen, Last Seen Compare if field values are subset Aliases, Goals, Secondary Motivations Threat Actor Source De-duplication Type of Deduplication Fields Compare even if fields are blank Name Compare if fields are not blank Description, Sophistication, Resource Level, Primary Motivation, First Seen, Last Seen Compare if field values are subset Aliases, Threat Actor Types, Threat Actor Roles, Personal Motivations, Secondary Motivations, Goals Threat Note Source De-duplication Type of Deduplication Fields Compare even if fields are blank Content Compare if fields are not blank Abstract Compare if field values are subset Authors Tool Source De-duplication Type of Deduplication Fields Compare even if fields are blank Name Compare if fields are not blank Description, Tool version Compare if field values are subset Tool Types, Attack Phases, Aliases Campaign Source De-duplication Type of Deduplication Fields Compare even if fields are blank Name Compare if fields are not blank Description, First Seen, Last Seen Compare if field values are subset Aliases, Objective Infrastructure Source De-duplication Type of Deduplication Fields Compare even if fields are blank Name Compare if fields are not blank Description, First Seen, Last Seen Compare if field values are subset Aliases, Attack Phases, Infrastructure Types Observed Data Source De-duplication Type of Deduplication Fields Compare even if fields are blank First Observed, Last Observed, Observed Count Compare if fields are not blank - Compare if field values are subset - Object Sighting Source De-duplication Type of Deduplication Fields Compare even if fields are blank Is Summary, Is Internal Compare if fields are not blank First Seen, Last Seen, Description, Count Compare if field values are subset - Threat Grouping Source De-duplication Type of Deduplication Fields Compare even if fields are blank Context, Name Compare if fields are not blank Description Compare if field values are subset - Threat Report Source De-duplication Type of Deduplication Fields Compare even if fields are blank Published, Name Compare if fields are not blank - Compare if field values are subset - Malware Source De-duplication Type of Deduplication Fields Compare even if fields are blank Name Compare if fields are not blank Description, First Seen, Last Seen Compare if field values are subset Aliases, Malware Types, Attack Phases, Process Architectures, Implementation Languages, Malware Capabilities Malware Analysis Source De-duplication Type of Deduplication Fields Compare even if fields are blank Product, Operating System, Host VM Compare if fields are not blank Version, Modules, Configuration Version, Analysis Definition Version, Analysis Engine Version, Result Name, Result Compare if field values are subset - Marking Definition Source De-duplication Type of Deduplication Fields Compare even if fields are blank Name, Definition Type, Definition Compare if fields are not blank - Compare if field values are subset - Data Source Source De-duplication Type of Deduplication Fields Compare even if fields are blank Name Compare if fields are not blank Description Compare if field values are subset Platforms, Collection Layers, Contributors Data Component Source De-duplication Type of Deduplication Fields Compare even if fields are blank Name, Data Source Compare if fields are not blank Description Compare if field values are subset - Related KB Please refer to the following knowledge base articles for detailed information regarding other elements of TISC processing workflow:KB1587754: This KB article documents the unique identification keys used for different threat intelligence entities within the TISC system. These keys are utilized during the Parent Identification phase, which occurs prior to the De-duplication process.KB1587758: KB article provides details regarding the aggregation logic implemented in the TISC Processing layer, which is executed immediately following the De-duplication phase.