sys_cluster_state table contains duplicate records with same system_id - Known Error

sys_cluster_state table contains duplicate records with same system_id Description The sys_cluster_state table can contain duplicate records with the same system_id. This can result in a node being incorrectly identified as offline and can result in the deletion of child jobs.

Only one record should be present in sys_cluster_state for a system_id.

DB connection being unhealthy is identified as the root cause for duplicate records.

Steps to Reproduce

1. Go to a record in sys_cluster_state 2. do Insert and Stay

On a real instance, it seems to occur when ClusterFactory is unable to get sys_cluster_state record likely when DB/Connection is unhealthy

Workaround As a workaround, add the following two unique indexes on sys_cluster_state

1. system_id 2. node_id

You can run the attached script to create the above unique indexes.