Product knowledge of Instance Data replication - Support and Troubleshooting

Product knowledge of Instance Data replication Summary .ns-kb-css-body-editor-container { p { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } span { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } h2 { font-size: 24pt; font-family: Lato; color: var(--now-color--text-primary, black); } h3 { font-size: 18pt; font-family: Lato; color: var(--now-color--text-primary, black); } h4 { font-size: 14pt; font-family: Lato; color: var(--now-color--text-primary, black); } a { font-size: 12pt; font-family: Lato; color: var(--now-color--link-primary, #00718F); } a:hover { font-size: 12pt; color: var(--now-color--link-primary, #024F69); } a:target { font-size: 12pt; color: var(--now-color--link-primary, #032D42); } a:visited { font-size: 12pt; color: var(--now-color--link-primary, #00718f); } ul { font-size: 12pt; font-family: Lato; } li { font-size: 12pt; font-family: Lato; } img { display: ; max-width: ; width: ; height: ; } } 1. OVERVIEW:

Instance Data Replication (IDR) is ServiceNow's built-in mechanism for automatically synchronizing data between two or more connected ServiceNow instances. Rather than requiring administrators to export and import data manually, IDR continuously watches for changes on a source instance and delivers them to one or more target instances in near real-time.

IDR is designed for organizations that operate multiple instances and need data to stay consistent across them - for example, a global company that has regional instances sharing a common catalog, or a production instance that seeds data to a lower environment. It is an asynchronous system, meaning changes are queued and delivered reliably even if the target instance is temporarily unavailable.

Common use cases include:

Keeping reference data (catalogs, configuration items, user groups) in sync across instances Seeding a new instance with baseline data from an existing one Distributing workloads across regional instances, where each region handles its own subset of records Maintaining a data archive or reporting instance that mirrors production records

2. CORE CONCEPTS: Before diving into the flow, it helps to understand the four fundamental building blocks of IDR.

Producer The instance that owns and originates the data. It watches for changes and pushes them outward. A Producer Replication Set defines exactly what data to share - which tables, which fields, which records.

Consumer The instance that receives and applies data changes from the Producer. A Consumer Replication Set defines where to receive the data and connects back to the Producer It subscribes to.

Entry Set A container that groups one or more table definitions together. Both Producer and Consumer reference the same Entry Set, which ensures they agree on what is being replicated.

Replication Entry An individual table configuration within an Entry Set. It specifies the exact table name, which fields to include, any filter conditions to limit which records qualify, and whether attachments or journal fields (work notes, comments) should also be replicated.

3. REPLICATION MODES IDR supports three distinct modes of operation:

Unidirectional Replication Data flows in one direction only — from Producer to Consumer. Changes made on the Consumer side to replicated records are not sent back. This is the simplest and most common mode.

Bidirectional Replication Both instances can produce and consume changes to the same data. When a record is changed on either side, the change is sent to the other. IDR tracks version numbers to prevent changes from bouncing back and forth indefinitely.

Discrete Replication A specialized mode where individual records are routed to specific consumers based on a field value. For example, incidents belonging to a specific assignment group might be routed only to the instance responsible for that group. Each record goes to exactly one consumer, which prevents duplication across targets.

4. END-TO-END DATA FLOW Phase 1 — Configuration and Setup An administrator on the source instance creates a Producer Replication Set and links it to an Entry Set that defines which tables and fields to share. On the target instance, an administrator creates a Consumer Replication Set that points back to the producer and references the same Entry Set.

Once both sides are configured, the two instances exchange an encrypted shared key that will be used to authenticate and verify every message exchanged between them. This negotiation happens automatically when the consumer is first activated.

Phase 2 — Initial Seeding When a Consumer is activated for the first time (or when an administrator manually triggers a seed), IDR performs a full data load from the producer table to the consumer table. This is called seeding.

The Producer reads all qualifying records, serializes them into replication payloads, and splits them into manageable chunks. These chunks are sent through the messaging layer (Hermes/Kafka topics). The Consumer receives the chunks, reassembles them, and inserts or updates the records in its target table. Progress is tracked throughout, and the operation can be paused, resumed, or cancelled.

Seeding can also be scheduled to repeat automatically at fixed intervals (every 6 or 12 hours) to periodically refresh the consumer's data.

Phase 3 — Delta (Ongoing) Replication Once seeding is complete and the Consumer enters the Active state, IDR switches to continuous delta replication. Any time a record is inserted, updated, or deleted on the source table, an event is generated and published to a dedicated Kafka topic. The Consumer subscribes to this topic and receives the event, then applies the change to its local copy of the data.

Delta replication is event-driven and happens within seconds under normal conditions. The topic naming includes the instance ID and replication set ID to ensure messages are isolated per subscription.

Phase 4 — Monitoring and Health Both sides continuously exchange heartbeat messages to confirm the connection is alive. Metrics such as how many messages are waiting in the queue (message lag), the time since the last message was received (data lag), and network response time (network lag) are collected and surfaced in the IDR dashboard. Administrators can set alert thresholds on each of these metrics.

5. KEY STATES AND WHAT THEY MEAN

A replication set moves through a defined set of states during its lifecycle. Understanding these states is essential for troubleshooting.

State Meaning Draft Created but not yet activated Activation Requested Admin triggered activation; system is negotiating the connection Seeding Initial data load is in progress Seeding Paused Seeding was paused by the user or due to an error Active Replication is running normally; delta changes are flowing Paused Replication was temporarily stopped by an admin Stopped Replication deactivated; can be restarted Error A failure occurred, and replication has halted. Requires admin attention Deleted The replication set was permanently removed

A seeding request has its own states:

Seeding State Meaning Queued Scheduled but has not started Started Actively running Completed Finished successfully Failed Could not complete Errored Interrupted by an error Cancelled Admin cancelled the operation Paused Paused mid-run

6. SECURITY AND AUTHENTICATION:

Shared Key Encryption

When a consumer first subscribes to a producer, the two instances negotiate a shared cryptographic key. Every message sent between them is signed using this key. If a message arrives with an unrecognized or expired key, it is rejected. Keys have expiry dates and are rotated over time. They are stored securely in encrypted form.

Role-Based Access

The idr_admin role is required to create, modify, or delete replication sets. The idr_subscriber role is required on the consumer instance to approve incoming subscription requests. All background replication jobs run under a dedicated system user account. Consumer Approval

When a producer first reaches out to establish a subscription, the consumer can be configured to require explicit admin approval before accepting. Until approved, replication does not begin. The approval status is tracked as Pending, Approved, or Denied.

7. IMPORTANT TABLES:

The following tables are central to how IDR operates. Understanding what each table stores is critical for diagnosing issues.

Configuration Tables

Table Purpose idr_replication_set Base table for all replication sets, shared fields for both producers and consumers idr_producer_set Extends idr_replication_set; represents the source side configuration idr_consumer_set Extends idr_replication_set; represents the target side configuration; holds the producer's URL and location idr_entry_set Defines a named group of replication entries shared between producer and consumer idr_replication_entry Defines per-table replication rules: which table, which fields, filter conditions, transformations idr_discrete_field_value For discrete replication, maps that route field values to the consumer

Runtime and Status Tables

Table Purpose idr_subscription Represents an active consumer-to-producer subscription; tracks both the consumer's status and the producer's view of that consumer idr_replication_status Full history of state transitions for a replication set, including timestamps and messages idr_inbound_info Consumer-side metrics: messages remaining in queue, last message arrival time, data lag idr_outbound_info Producer-side metrics: last message sent time and last dispatched message ID

Seeding Tables:

Table Purpose idr_seeding_request Represents a seeding operation (initial, scheduled, or DCT-triggered), tracks status, duration, and percent complete idr_seeding_detail Tracks per-table progress within a seeding request: records sent and records succeeded idr_scheduled_replication_request Stores configuration for periodic re-seeding, including schedule interval and next run time

Messaging Tables:

Table Purpose idr_chunk Temporary storage for large payloads split into multiple message chunks; reassembled when all chunks arrive dr_entry_schema Stores serialized schema for replication entries, used to verify compatibility between producer and consumer

Error and Monitoring Tables:

Table Purpose idr_error_tracker Tracks errors that need admin attention; each record has an error state of Unresolved or Resolved idr_replication_payload_error Stores the exact content of a payload that failed to process, the error message, and which record was affected idr_replication_log Full audit trail of all replication activity; every message received, sent, or processed is logged here idr_retry_request Tracks out-of-order message recovery requests; records cursor positions where the error started and ended idr_system_status Overall IDR system health indicator idr_observability_metric Collected performance and health metrics for monitoring dashboards idr_queue_stats Queue processing performance statistics per replication set

Security Tables:

Table Purpose idr_replication_shared_key Stores the encrypted shared key used to sign and verify messages for each subscription

Data Comparison Tables:

Tables Purpose idr_instance_comparison_request Initiates a data comparison between producer and consumer to identify gaps or mismatches

8. SCHEDULED JOBS - COMPLETE REFERENCE:

IDR depends on a set of background scheduled jobs to drive replication, seeding, heartbeats, monitoring, and error recovery. If any of these jobs are disabled or fail silently, the corresponding IDR function stops working - often without an obvious error surfacing to the end user. This section describes every job, what it does, how often it runs, which side of the replication it runs on, and what breaks if it is disabled.

8.1 CORE DELTA REPLICATION JOBS These are the most critical jobs. They are responsible for the actual movement of data changes between instances on an ongoing basis.

IDR Producer Job Runs on: Producer (source) instance Frequency: Continuously in short batches; maximum 15 minutes per execution

What it does: This is the engine of ongoing delta replication. It continuously polls the internal data change queue for new inserts, updates, and deletes that have occurred on replicated tables. For each change it finds, it transforms the record according to the replication entry rules (applying field filters, discrete routing, and transformations), then publishes the resulting payload to the appropriate Hermes topic so consumers can receive it. It tracks a cursor position in the queue so that if the job is interrupted, it resumes from where it left off rather than re-sending already-processed changes. It also enforces bidirectional loop prevention if a change originated from a consumer (not the producer), it is not resent back.

If disabled: Data changes on the producer stop being published. Consumer instances will no longer receive any updates. The replication will appear Active in status, but data will silently diverge.

IDR Delta Consumer Job Runs on: Consumer (target) instance Frequency: Continuously in short batches

What it does: Listens to the Hermes topic for incoming delta messages published by the producer. For each message it reads, it deserializes the payload, identifies the target table and operation type (insert, update, or delete), and applies the change to the local table. It advances a cursor position as messages are successfully applied so that progress is not lost on restart. A mutual exclusion (mutex) lock prevents multiple app nodes from processing the same messages simultaneously.

If disabled: Incoming delta messages pile up in the message queue unprocessed. The consumer falls further behind over time. Replication lag grows until the subscription eventually times out or enters an error state.

IDR Hermes Delta Consumer Job Runs on: Consumer instance (Hermes-connected instances only) Frequency: Continuously in short batches

What it does: Performs the same function as the IDR Delta Consumer Job but is specifically optimized for instances using the Hermes V2 messaging infrastructure. It handles Hermes-native topic subscription, message deserialization, and consumer group offset management. Instances that have been upgraded to Hermes use this job instead of the legacy Kafka consumer.

If disabled: Same impact as IDR Delta Consumer Job on Hermes-enabled instances.

IDR Metadata Consumer Job / IDR Hermes Metadata Consumer Job Runs on: Consumer instance Frequency: Continuously in short batches

What it does: Processes control and coordination messages that accompany data replication - these are not data records themselves but instructions such as seeding progress updates, heartbeat acknowledgments, shared key exchanges, activation confirmations, and out-of-order retry signals. Without this job, all the coordination messages that manage the state of a replication relationship go unprocessed.

If disabled: Replication set state transitions stop working. Activations, seeding completions, key exchanges, and heartbeat responses are never applied. The replication appears stuck in intermediate states.

8.2 SEEDING JOBS

These jobs manage the initial bulk data transfer and all subsequent seeding operations.

IDR Seeding Producer Job Runs on: Producer instance Frequency: Triggered on demand (when a seeding request is queued); maximum 15 minutes per execution

What it does: When a consumer is newly activated, when an administrator manually triggers a seed, or when the Data Comparison Tool requests remediation seeding, this job handles the bulk send from the producer side. It reads all qualifying records from the source table in large batches, serializes each record into a replication payload, splits oversized payloads into chunks, and publishes them to a dedicated seeding topic (separate from the delta topic). It tracks how many records have been sent and supports pause and cancel signals from operators. If a seeding run is interrupted, it resumes from the last successfully sent record on the next execution.

If disabled: No baseline data is ever delivered to new subscribers. Consumers are activated but receive only future deltas, meaning historical data is missing entirely.

IDR Seeding Consumer Job Runs on: Consumer instance Frequency: Triggered when seeding chunks arrive in the seeding topic

What it does: Receives the seeding chunks published by the IDR Seeding Producer Job, reassembles them from the idr_chunk table when they span multiple messages, deserializes the full record payload, and performs an upsert (insert or update) on the local target table for each record. It tracks per-table progress in idr_seeding_detail so that partially completed seeding can resume rather than restart. Once all records for all tables in the entry set have been received, it marks the seeding request as Completed and transitions the replication set to Active state.

If disabled: Seeding chunks arrive on the consumer but are never applied. Seeding requests stay in Started state indefinitely. The consumer never transitions to Active.

IDR Schedule Seeding Queuing Job Runs on: Producer instance Frequency: Every 5 minutes

What it does: Checks all producer replication sets that have scheduled seeding configured. For each set whose next scheduled run time has been reached or passed, it scans the change queue for records within the configured time window, calculates how many records each table needs to sync, creates a Data Comparison Transaction (DCT) request for each active subscriber, and updates the next scheduled run time on the producer set record. Sets that use bidirectional or discrete replication are kipped as scheduled seeding does not apply to them.

If disabled: Scheduled seeding never triggers. Periodic data refresh does not happen. Consumers may gradually accumulate data drift if delta messages were ever missed.

IDR Schedule Seeding Processor Job Runs on: Producer instance Frequency: Every 1 minute; maximum 15 minutes per execution

What it does: Picks up DCT requests that were queued by the IDR Schedule Seeding Queuing Job and processes them by scanning the change data queue in large batches to extract the exact record IDs to be synced. It inserts those record IDs into the comparison result table marked as mismatched, which queues them for auto-seeding to the consumer. Processing is check pointed regularly so that if the job hits its runtime limit, it saves its position and picks up exactly where it left off on the next run.

If disabled: Queued scheduled seeding DCT requests are never processed, so scheduled seeding records are never actually sent to consumers.

9. QUICK REFERENCE: WHERE TO LOOK WHEN SOMETHING GOES WRONG

Symptom First Table to Check Second Table to Check Replication won't activate idr_error_tracker idr_replication_log Seeding failing idr_seeding_request idr_replication_payload_error Records missing from consumer idr_replication_payload_error Run DCT via idr_instance_comparison_request Delta changes not arriving idr_inbound_info (check lag values) idr_subscription (check consumer status) Unknown key errors idr_replication_shared_key idr_replication_payload_error Out-of-order errors idr_retry_request idr_replication_log Scheduled seeding not running idr_scheduled_replication_request Check scheduled job status (sys_trigger and sysauto_script) General health check idr_system_status idr_observability_metric

10. HERMES AND IDR - ROLE, INTEGRATION, AND ISSUE ISOLATION

10.1 WHAT HERMES IS:

Hermes is ServiceNow's internal event streaming platform built on top of Apache Kafka. Rather than exposing raw Kafka to every feature that needs messaging, Hermes wraps Kafka with a managed layer that handles multi-datacenter cluster configuration, topic lifecycle management, authentication, and failover. Think of Hermes as the "post office" inside the ServiceNow platform - it manages all the infrastructure needed to reliably deliver messages between services and instances without each consumer needing to know where the brokers physically live.

For IDR, Hermes is the transport layer. Every data change that IDR replicates - every insert, update, delete, seeding chunk, and control message - travels through a Hermes - managed Kafka topic before reaching the consumer instance. IDR does not connect directly to raw Kafka in modern deployments; it goes through Hermes.

Hermes configuration lives in the hermes_cluster_config table on each instance. This table tells IDR (and the rest of the platform) which Kafka broker addresses to use, which datacenter they belong to, and which cluster is the primary for a given datacenter. If this table is empty or has stale entries, IDR cannot connect to the messaging layer at all.

10.2 HOW IDR USES HERMES:

IDR uses Hermes for every phase of its operation. There is no phase in which data moves between instances without going through a Hermes topic, except one time, which is when the initial subscription record is sent via REST API call from the consumer to producer Instance

IDR creates three categories of Hermes topics:

Delta Topics Purpose: Carry incremental data changes (inserts, updates, deletes) from the producer instance to the consumer instance on an ongoing basis. Naming: Built from the producer instance ID, consumer instance ID, and replication set ID combined into a unique topic name. Direction: One-way, producer to consumer. Lifetime: Permanent for the duration of the replication subscription.

Metadata Topics Purpose: Carry control and coordination messages between the producer and consumer. These are not data records- they carry instructions such as activation confirmations, seeding requests, shared key exchanges, heartbeat signals, upgrade coordination messages, and out-of-order retry signals. Naming: Built from the producer instance ID and consumer instance ID. Direction: Bidirectional — both sides publish to and consume from these topics. Lifetime: Permanent for the duration of the replication subscription.

Seeding Topics Purpose: Carry the bulk data payload during initial seeding or scheduled re-seeding. These topics are separate from delta topics to prevent seeding traffic from interfering with ongoing delta replication. Naming: Built from the consumer instance ID, sometimes with a timestamp suffix for enhanced seeding operations. Direction: One-way, producer to consumer. Lifetime: May be ephemeral — cleaned up after seeding completes.

All three topic types are created automatically by IDR when a replication set is activated. IDR calls into the Hermes topic management layer to create them on the correct cluster. If topic creation fails, the replication set does not activate.

Release .ns-kb-css-body-editor-container { p { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } span { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } h2 { font-size: 24pt; font-family: Lato; color: var(--now-color--text-primary, black); } h3 { font-size: 18pt; font-family: Lato; color: var(--now-color--text-primary, black); } h4 { font-size: 14pt; font-family: Lato; color: var(--now-color--text-primary, black); } a { font-size: 12pt; font-family: Lato; color: var(--now-color--link-primary, #00718F); } a:hover { font-size: 12pt; color: var(--now-color--link-primary, #024F69); } a:target { font-size: 12pt; color: var(--now-color--link-primary, #032D42); } a:visited { font-size: 12pt; color: var(--now-color--link-primary, #00718f); } ul { font-size: 12pt; font-family: Lato; } li { font-size: 12pt; font-family: Lato; } img { display: ; max-width: ; width: ; height: ; } } All