SAM Reconciliation Causes MariaDB Replication Lag Due to Mass Updates on cmdb_sam_sw_install - Known Error

SAM Reconciliation Causes MariaDB Replication Lag Due to Mass Updates on cmdb_sam_sw_install Description Short Description: During SAM Software License Reconciliation, SamPublisherCalculator.reconcileLeftoverEntities issues bulk UPDATE statements on cmdb_sam_sw_install affecting 100k–200k rows per product, resulting in slow DMLs and replication lag on MariaDB replicas.

Problem: When the SAM - Software License Reconciliation Worker scheduled job runs, the method SamPublisherCalculator.reconcileLeftoverEntities calls updateMultiple() to set is_reconciled , product_result , and related fields on all install records matching a given norm_publisher / norm_product combination. For some large publishers (e.g. Microsoft), this can affect 100,000–200,000 rows per product in a single statement.

On MariaDB instances with read replicas, the resulting DML is applied serially on the replica (via slave worker threads), causing the replica to fall behind primary — observed as replication lag. The issue is persistent and recurs across every reconciliation run while the affected software models are in scope.

Root Cause:

SamPublisherCalculator.reconcileLeftoverEntities uses updateMultiple() without row-level batching. For publishers with large install footprints, this produces unbounded bulk updates that:

Hold row locks for the entire duration of the update, blocking concurrent writes. Generate large binary log events that replica threads must replay sequentially, creating lag. The fix involves introducing chunked/batched updates to limit the number of rows modified per statement, reducing both lock duration and binlog event size.

Steps to Reproduce

On an instance running SAM reconciliation with a large software publisher footprint, observe slow UPDATE statements on cmdb_sam_sw_install during the SAM - Software License Reconciliation Worker job run.

Example slow DML observed (runtime ~23 min 56 sec on a single statement)

UPDATE cmdb_sam_sw_install SET is_reconciled = 1, product_result = ' ' WHERE ( (norm_publisher = ' ' AND norm_product = ' ' AND inferred_suite_product IS NULL) OR inferred_suite_product = ' ' ) AND product_result IS NULL AND license_metric_result IS NULL AND software_model_result IS NULL;

Workaround None necessary. The fix should be present in all in-band releases.