Instance Data Replication (IDR) – Replication Set Processing Order, Queue Management, and Throughput Limitations - Support and Troubleshooting

Instance Data Replication (IDR) – Replication Set Processing Order, Queue Management, and Throughput Limitations Issue .ns-kb-css-body-editor-container { p { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } span { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } h2 { font-size: 24pt; font-family: Lato; color: var(--now-color--text-primary, black); } h3 { font-size: 18pt; font-family: Lato; color: var(--now-color--text-primary, black); } h4 { font-size: 14pt; font-family: Lato; color: var(--now-color--text-primary, black); } a { font-size: 12pt; font-family: Lato; color: var(--now-color--link-primary, #00718F); } a:hover { font-size: 12pt; color: var(--now-color--link-primary, #024F69); } a:target { font-size: 12pt; color: var(--now-color--link-primary, #032D42); } a:visited { font-size: 12pt; color: var(--now-color--link-primary, #00718f); } ul { font-size: 12pt; font-family: Lato; } li { font-size: 12pt; font-family: Lato; } img { display: ; max-width: ; width: ; height: ; } } Users require clarity on the processing order of Replication Sets in Instance Data Replication (IDR), reasons for variable processing rates across sets, management of large inbound queue buildups, and the throughput limitations that govern IDR data processing.

Symptoms .ns-kb-css-body-editor-container { p { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } span { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } h2 { font-size: 24pt; font-family: Lato; color: var(--now-color--text-primary, black); } h3 { font-size: 18pt; font-family: Lato; color: var(--now-color--text-primary, black); } h4 { font-size: 14pt; font-family: Lato; color: var(--now-color--text-primary, black); } a { font-size: 12pt; font-family: Lato; color: var(--now-color--link-primary, #00718F); } a:hover { font-size: 12pt; color: var(--now-color--link-primary, #024F69); } a:target { font-size: 12pt; color: var(--now-color--link-primary, #032D42); } a:visited { font-size: 12pt; color: var(--now-color--link-primary, #00718f); } ul { font-size: 12pt; font-family: Lato; } li { font-size: 12pt; font-family: Lato; } img { display: ; max-width: ; width: ; height: ; } } Replication Sets show uneven data processing volumes (some sets process large amounts of data while others process minimal data) Large inbound queue buildup (up to 570K records) observed, with slow clearing initially followed by rapid drops Uncertainty about whether IDR has specific time windows for reduced processing Questions about maximum data volumes IDR can handle per day Concern that some Replication Sets may be prioritized over others Release .ns-kb-css-body-editor-container { p { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } span { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } h2 { font-size: 24pt; font-family: Lato; color: var(--now-color--text-primary, black); } h3 { font-size: 18pt; font-family: Lato; color: var(--now-color--text-primary, black); } h4 { font-size: 14pt; font-family: Lato; color: var(--now-color--text-primary, black); } a { font-size: 12pt; font-family: Lato; color: var(--now-color--link-primary, #00718F); } a:hover { font-size: 12pt; color: var(--now-color--link-primary, #024F69); } a:target { font-size: 12pt; color: var(--now-color--link-primary, #032D42); } a:visited { font-size: 12pt; color: var(--now-color--link-primary, #00718f); } ul { font-size: 12pt; font-family: Lato; } li { font-size: 12pt; font-family: Lato; } img { display: ; max-width: ; width: ; height: ; } } Applicable to all IDR implementations across supported releases.

Cause .ns-kb-css-body-editor-container { p { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } span { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } h2 { font-size: 24pt; font-family: Lato; color: var(--now-color--text-primary, black); } h3 { font-size: 18pt; font-family: Lato; color: var(--now-color--text-primary, black); } h4 { font-size: 14pt; font-family: Lato; color: var(--now-color--text-primary, black); } a { font-size: 12pt; font-family: Lato; color: var(--now-color--link-primary, #00718F); } a:hover { font-size: 12pt; color: var(--now-color--link-primary, #024F69); } a:target { font-size: 12pt; color: var(--now-color--link-primary, #032D42); } a:visited { font-size: 12pt; color: var(--now-color--link-primary, #00718f); } ul { font-size: 12pt; font-family: Lato; } li { font-size: 12pt; font-family: Lato; } img { display: ; max-width: ; width: ; height: ; } } Questionaire

Resolution .ns-kb-css-body-editor-container { p { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } span { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } h2 { font-size: 24pt; font-family: Lato; color: var(--now-color--text-primary, black); } h3 { font-size: 18pt; font-family: Lato; color: var(--now-color--text-primary, black); } h4 { font-size: 14pt; font-family: Lato; color: var(--now-color--text-primary, black); } a { font-size: 12pt; font-family: Lato; color: var(--now-color--link-primary, #00718F); } a:hover { font-size: 12pt; color: var(--now-color--link-primary, #024F69); } a:target { font-size: 12pt; color: var(--now-color--link-primary, #032D42); } a:visited { font-size: 12pt; color: var(--now-color--link-primary, #00718f); } ul { font-size: 12pt; font-family: Lato; } li { font-size: 12pt; font-family: Lato; } img { display: ; max-width: ; width: ; height: ; } } Replication Set Processing Order

IDR does not prioritize Replication Sets. Instead, processing order is determined by chronological record modification timestamps:

On the producer side, messages are processed in chronological order with the oldest changes sent first If 100,000 records in Replication Set A are modified, followed by 5,000 records in Replication Set B, the producer will finish sending the 100k records from Set A before beginning to send the 5k records from Set B This sequential producer behavior means large data bursts are not ideal for IDR, as they can delay other Replication Sets Consumer-Side Processing

On the consumer side, multiple parallel Consumer jobs can process data from different streams simultaneously:

If Replication Set A (100k records) is being produced first, those records enter the inbound queue first Once Replication Set B records become available in their topic (i.e., after Set A production begins but while records still queue), the consumer can process both streams in parallel if multiple Consumer jobs are running (default behavior) Actual consumption speed depends on available Consumer job capacity and instance resource availability Large Inbound Queue Buildups – Causes and Expected Behavior

Typical Scenario: Inbound queue reaches 570K+ records, remains high while slowly decreasing, then drops massively over time.

Root Cause — Data Bursts:

This behavior is expected and normal. IDR is designed to stream a moderate amount of data continuously over time. When a large number of records are updated simultaneously (a "data burst"), the system experiences a processing delay until records are eventually replicated:

Large data bursts (e.g., bulk updates, Discovery processes) create a temporary backlog The queue remains high because the production rate exceeds the consumption rate during the burst window As new data stops flowing in, the existing queue gradually clears The "massive drop" occurs when the queue has been depleted and consumption catches up to the current production rate Why This Occurs:

IDR relies on background jobs to process data; if the instance is heavily utilized, these jobs run less frequently (though this is typically not the primary cause of buildup) Large simultaneous updates (common in Discovery, bulk imports, or mass updates) overwhelm the replication pipeline IDR must work through the queue sequentially, which takes time proportional to the data volume This is not a system malfunction — it is working as designed.

Processing Limitations and Throughput Maximums

IDR enforces the following hard limits:

Limitation Value Maximum record size 10 MB per record Continuous replication throughput Approximately 1 million records per day (across all Replication Sets combined) Seeding duration Must complete within 7 days Important: The 1 million records per day limit is a combined total across all active Replication Sets, not per set.