CMDB Health Dashboard: Last updated value on the widgets is not being updated after the health jobs are run


Description

The last updated value on the correctness scorecard is not updating although the correctness scheduled job ran and it created results and tasks in the cmdb_health_result table.

Cause

The last updated value on the CMDB health dashboard for correctness scorecard, pulls the value from the "Started On" field of the cmdb_health_metric_status record for "Correctness" metric which has "active" flag set to true

https://<instancename>.service-now.com/cmdb_health_metric_status_list.do?sysparm_nostack=true&sysparm_query=&sysparm_first_row=1&sysparm_view= 

In the above list, if there are two entries for "Correctness" metric. The one with "active" flag set to false is the most current metric processor run. However the active flag will be set to false until the status becomes "Complete" from "Daily time out pause". 

The reason why it is in "Daily time out pause" is because of one of the sub-metrics under the "Correctness" did not complete its complete processing yet. 


Resolution

The "staleness", "orphan" and "duplicate" metrics are all sub-metrics under the "Correctness". So, the correctness metric score calculation would be dependent on these 3 sub-metrics.

Also there is a property "glide.cmdb.health.metricProcessor.maxRunningTime" that defines the max runtime for the metric processors. The default value is 2 hours 
https://<instancename>.service-now.com/nav_to.do?uri=sys_properties.do?sys_id=d5e8d89037101200f212cc028e41f118 

In certain cases where there are a large number of CI's the metric processing goes beyond 2 hours and in that case you will see the state set to "Daily time out pause". In the next scheduled run of the job, the metric processing picks up the context where it left the other day and continues the processing. 

Once the correctness job state is set to complete, you will be seeing the correct date in the dashboard.