Validating and comparing instance upgrades - Support and Troubleshooting

Validating and comparing instance upgrades Description Upgrading and patching your instance requires planning, testing, and validation. To ensure a safe and effective upgrade, read the release notes, create upgrade plans, and test your upgrade on non-production instances before upgrading your production instance.

ServiceNow provides guidelines on how to ensure a smooth upgrade but there can be scenarios where upgrade behaves differently in production instance compared to lower environment instance upgrades.

One of the most commonly reported issues is why an upgrade touched a customized file in one instance compared to another. This document provides an overall view of how the upgrade engine identifies customizations and what to look for when analyzing upgrades between instances.

'Customer Updates' table and the 'Recorded at' field Before we proceed with how an upgrade handles customizations, we need to dig a bit deeper on the 'Customer Updates' table.

The ' Customer Updates ' table is used by the upgrade engine to identify instance customizations. Every configuration change that is made in the system is recorded on the 'Customer Updates' [sys_update_xml] table chronologically.

[Please note: limited to out-of-the-box tables having " update_synch " attribute set as true]

Order of records in this table makes a difference in how the upgrade engine evaluates customizations. Prior to Jakarta, the updates were ordered based on 'Updated on' field however post-Jakarta 'Recorded at' field is used.

'Recorded at' field : From Jakarta onwards, a new field has been added to the 'Customer Updates' table called 'Recorded At'. This field has sub-millisecond resolution whereas the 'Updated on' has seconds resolution. So if you have two updates that are recorded in precisely the same second, ordering them by 'Updated on' field can result in customer updates being evaluated incorrectly. That's where 'Recorded at' field comes into the picture.

Along with 'Recorded at' field, 'Replace on Upgrade' field plays an important role in whether customization should be reverted or not.

'Replace on Upgrade' field : When you change any non-excluded fields on a record, a corresponding record is added in the Customer Update [sys_update_xml] table and the Replace on upgrade field is set to false.

Note: For non-excluded fields on a record, refer to ' Update set administration '. How the upgrade identifies customizations In order to understand whether a file will be touched by an upgrade or not, we need to understand how the upgrade engine logic works. Following is a quick rundown of how the upgrade engine identifies files that are customized vs which are not

The upgrade engine uses the 'Customer Updates' table to identify customizations. The upgrade engine queries this table similar to below where it ignores records coming from baseline update set or remote update set:

These records are sorted based on 'Recorded at' field (post-Jakarta) and 'Updated on' (pre-Jakarta) for each file The upgrade engine then checks the 'Replace on upgrade' flag on the most recent record for each file to decide whether the file should be touched by the upgrade or not. If the 'Replace on upgrade' is set as true, the file will be touched by the upgrade otherwise it will be ignored by the upgrade and the customizations remain intact. 'Upgrade History' module and 'Skipped Changes to Review' The Upgrade History module tracks every upgrade made to an instance. Administrators can use the module to resolve upgrade conflicts and optionally to revert customizations to base system versions to take advantage of new features.

An upgrade history record is created for each upgrade that is run. To view an upgrade history record, navigate to System Diagnostics > Upgrade History and click the upgrade. It summarises the following information:

Changes skipped Total number of records that were different from the previous upgrade but were skipped, mostly likely due to customization. Changes skipped is the sum of the records that have disposition of skipped manual merge (where the value of changed is true), added to the number of records that have disposition of skipped error, added to the number of records that were skipped and different.

Note: To prevent your customizations from being overwritten during system upgrades, the upgrade process skips (does not apply the update to) objects that have been customized. One of your responsibilities as the administrator is to resolve each update that was skipped due to a customization. To resolve a skipped update, you review the reason for each skipped record and then either merge the customization or revert the customization to the base system.

Changes applied Total number of the changes that were applied in this upgrade. Changes applied is sum of updated and different records, added to the number of deleted records (where the value of changed is true) added to the number of inserted records (where the value of changed is true).

Changes processed Total number of items processed during this upgrade. Changes processed is the sum of Changes skipped, plus Changes applied. This module should be the go-to place to review upgrades and the changes applied and skipped by the upgrade.

Upgrade troubleshooting: Why a record was touched by the upgrade? If you have noticed a certain file has been touched by the upgrade which it shouldn't have, the place to check would be the 'Customer Updates' table. Build a filter on this table, based on 'Name' of the file in question, ignoring records coming from the baseline update set or remote update set and ordered by 'Recorded at'. Below is an example of the query on 'Customer Updates' table for file name 'sys_script_725330019f22120047a2d126c42e70fa':

As you can see, the file's latest version where 'Recorded at' is set as 2018-04-14 02:04:39, the 'Replace on upgrade' field is set as 'true'. Given the most recent record is set as 'true' the upgrade will revert this file's customizations to its out-of-the-box state during the upgrade.

You can then perform the same steps in your other instances to verify if the same records are returned by the query above. Usually, where we do see discrepancies between instances is when these records do not match. In your lower environments, the 'Replace on upgrade' might have been reverted, say by a recent commit of an update set, hence the difference in the behavior of the upgrade.

Upgrade comparison between instances As part of the upgrade best practices, an upgrade of a production instance should always be tested in a lower environment. This lower environment should be a full clone of your production instance (including large tables and attachments).

However, post-upgrade of your production instance there can be differences reported in the number of 'Skipped Changes to Review' compared to what you saw in your sub-production instance.

Here are some step-by-step instructions on comparing upgrades between instances (say between Instance A and instance B where both were upgraded to the same version):

In Instance A, Go to System Diagnostics > Upgrade History and open the record for the upgrade in question. In the Related Lists, there is 'Upgrade Details' tab. Open this list in a new window by right-clicking the query and select "Open in new window" as shown below:

This opens the 'Upgrade Details' records for this particular upgrade in a new window. Filter out the results for 'Disposition' as 'Not Latest' Personalize list to only view File Name and Disposition columns Sort results by Created and File Name fields (in ascending order). It should look something like this:

Export the results records into an Excel file ( you might need to adjust export limits ) Do the same steps for export of Upgrade Details records for the other instance, Instance B's upgrade and copy the File Name and Disposition fields from there to the excel file created above but in a new sheet. We can then use VLOOKUP in this excel sheet to compare the dispositions between Instance A and Instance B. Sample excel file attached. We can use this file, to compare files that had different "Disposition" in the two upgrades being compared. What the excel sheet could look like is as follows (please note the files listed are dummy files):

Once you identify the records having a different disposition, review those files in their respective instances and validate that file in 'Customer Updates' table if the 'Replace on Upgrade' flag is set for the most recent record. Additional Information Please note the method stated above to compare instance upgrades is a manual process and would require review to identify if the upgrade updates were as you expected and why they differed.

Prepare your upgrade How to process the skipped records and all available dispositions: Upgrade Monitor module: Upgrade an individual instance