The pre-Zurich to Zurich upgrade of the 'sn_itam_health_chk' plugin can take a very long time - Known Error

The pre-Zurich to Zurich upgrade of the 'sn_itam_health_chk' plugin can take a very long time Description When upgrading from a pre-Zurich release (such as Yokohama or Xanadu) to any Zurich release, the ‘sn_itam_health_chk’ plugin upgrade process deletes two specific Instance Scan checks:

‘SAM - Stale VMs or Hosts consuming License’ ‘SaaS - M365 - Installation check’ Due to cascade rules in the ‘scan_finding’ table, all findings associated with these checks are also deleted during the upgrade. The upgrade time is directly proportional to the number of records in the ‘scan_finding’ table. At a rate of approximately 10 milliseconds per record, this translates to:

100,000 records: approximately 17 minutes 1,000,000 records: approximately 3 hours 10,000,000 records: approximately 28 hours 15,000,000 records: approximately 42 hours This issue affects only pre-Zurich to Zurich upgrades. Within-family Zurich upgrades (e.g., Zurich Patch 4 to Zurich Patch 4 Hotfix 3) are not affected. Instances which have already upgraded to Zurich Patch 1 or later will not encounter this issue on subsequent patch upgrades.

Steps to Reproduce

Identify an instance running a pre-Zurich release (Yokohama, Xanadu, or earlier) with the ‘sn_itam_health_chk’ plugin installed. Confirm the instance has a large number of records in the ‘scan_finding’ table (see Pre-Upgrade Validation below for how to check). Initiate an upgrade to any Zurich release. The upgrade will appear to stall on the ‘sn_itam_health_chk’ plugin update while the cascading record deletions are processed. Workaround Option 1: Pre-Upgrade Mitigation (Preferred Method - Before Starting the Upgrade) This is the recommended approach. Perform these steps before initiating the upgrade to prevent the extended upgrade time.

Check the scan_finding record count:

Run the following query to determine the number of records in the ‘scan_finding’ table: SELECT COUNT(*) FROM scan_finding; If the count exceeds 500,000 records, the upgrade time may be significantly extended. Use the estimates in the Description section above to project the additional upgrade time.

Implement a table cleaner rule:

Configure a table cleaner rule in the Instance Scan application to periodically remove stale ‘scan_finding’ entries. This rule should target findings associated with the following checks: ‘SAM - Stale VMs or Hosts consuming License’ ‘SaaS - M365 - Installation check’

Run the table cleaner:

Execute the table cleaner rule to remove stale entries before initiating the upgrade. Verify the ‘scan_finding’ record count has been reduced to an acceptable level.

Proceed with the upgrade:

Once the ‘scan_finding’ table has been cleaned, proceed with the upgrade. The ‘sn_itam_health_chk’ plugin update should complete in a reasonable timeframe. Option 2: Mid-Upgrade Relief (If the Issue Is Encountered During an Active Upgrade) If the upgrade has already been initiated and is stalled on the ‘sn_itam_health_chk’ plugin update, contact ServiceNow Support unless the instance is self-hosted.

The procedure below involves direct database operations. This can only be performed by ServiceNow Support engineers (or a DBA for self-hosted instances) via a Change Request and requires customer approval. It is recommended to test on a sub-production instance first.

Validate on a sub-production instance (recommended):

If a sub-production instance is available with the same upgrade stalled on the same plugin, execute the procedure there first to confirm successful completion.

Rename the existing 'scan_finding' table (requires a Change Request):

Connect to the database and execute the following commands:

BEGIN; ALTER TABLE scan_finding RENAME TO scan_finding_old; CREATE TABLE scan_finding (LIKE scan_finding_old INCLUDING ALL); COMMIT;

This renames the existing ‘scan_finding’ table (with all its records) to ‘scan_finding_old’ and creates a new empty ‘scan_finding’ table with an identical structure, including all indexes and constraints.

Verify:

Confirm the new table is empty and the backup table contains the original records:

SELECT COUNT(*) FROM scan_finding; -- Should return 0 SELECT COUNT(*) FROM scan_finding_old; -- Should return the original record count

Monitor the upgrade:

The ‘sn_itam_health_chk’ plugin upgrade should complete within minutes following this action. Continue to monitor the overall upgrade progress, including MID Server upgrades.

Post-upgrade cleanup: After the upgrade is complete: * Run Instance Scan to repopulate the ‘scan_finding’ table with current data. * The ‘scan_finding_old’ backup table can be retained for reference or removed after confirming the upgrade completed successfully and Instance Scan has repopulated the data. * Implement a table cleaner rule (see Option 1, Step 2) to prevent future accumulation of stale records. Related Problem: PRB1893349