Instance upgrade FAQ - Frequently Asked QuestionsIssue This article provides answers to the most popular questions about instance upgrades as well as relief for common related issues. This information applies to out-of-box instances. Make sure to review the links under each section for access to the most information, as customizations present in your instance can affect the listed behaviours. SELF-HOSTED/On-Premise customers, please refer to this KB as well KB0598275 - Upgrade Best Practices for Self-Hosted Customers 1. How to Schedule/Modify/Cancel the upgrade Change in ServiceNow Support?2. Patching program and End of Life upgrade program KB's3. What is the best practice to be followed for upgrades?4. Why didn't my upgrade start and why did it go past the planned start time on the ServiceNow Support change?5. Which is the best process to DRY RUN my upgrade?6. For how long will my upgrade run?7. What happens in my instance during an upgrade?8. Can we take an ad-hoc backup prior to the upgrade?9. Which are my options to recover my data in case of Data corruption after an upgrade?10. What are the features impacted on the instance/application during an upgrade?11. How to restrict users during an upgrade?12. Can ServiceNow Tech Support monitor my upgrade?13. What to do when the node upgrade has failed/node is RED in Upgrade Summary?14. Can I roll back/Downgrade an upgrade?15. How to use an Update Set to capture reverted customizations made after an upgrade?16. Why is the skipped record count different from my SUB PROD instance?17. Why is there a skipped updates count mismatch between the Upgrade Summary and the Upgrade History records?18. Can we cancel the upgrade while it is in progress?19. How do I review Skipped Update Records and track its resolution? (new Paris & after feature)20. How to notify upcoming upgrade on your P+instance21. Revert customized record to OOB and transfer via Update-set for Upgradeability22. How to ensure workarounds applied are overwritten with fixed OOB records during the upgrade23. Can I view an upgrade Summary Report for an older upgrade? 1. How do you Schedule/Modify/Cancel upgrades for your instances using ServiceNow Support? Please see KB0541128 - How to manage and schedule instance upgrades Missing timeslots for instance Upgrade 2. Patching program and End of Life upgrade program KB's KB0696901 - ServiceNow Patching Program FAQsKB0610454 - End of Life aka Unsupported Release Family Upgrades FAQ 3. What is the best practice to be followed for upgrades? This chapter of our documentation explains the best practices to be followed for an application upgrade: Upgrade your instance. You might also be interested in this video on our YouTube channel Upgrading to a New Release 4. Why didn't my upgrade start and why did it go past the planned start time on the ServiceNow Support change? The planned start of the ServiceNow Support change determines when the Assigned WAR version for the instance should be changed in the ServiceNow instance record. The upgrade is triggered by the "Upgrade/Check distribution for possible upgrade" Job on the instance which runs in an hourly interval(OOB) and this job checks the ServiceNow instance record every hour for the Assigned WAR version. When it finds a new Assigned WAR version, it will download the new WAR version. Note: Instance upgrade can take up to 1 hour to trigger on the instance from the planned start time of the ServiceNow Support Change. The actual Work start time is updated on the ServiceNow Support Change when the actual upgrade is triggered on the instance. You can check the upgrade jobs by going to this URL on your instance ( Pre Paris): /sys_trigger_list.do?sysparm_query=name%3Dupgrade ( Paris+): /sys_trigger_list.do?sysparm_query=nameLIKEpossible%20upgrade&sysparm_view= From Paris "Upgrade" job is replaced by "Check distribution for possible upgrade" "Check Upgrade Script" job is replaced by "Check database for possible upgrade" Please make sure the value on the System ID is blank on the "Upgrade/Check distribution for possible upgrade" Job. If there is a value, please make sure that this is selected to --None-- Hardcoding the System ID is not a recommended approach and there are chances that the node selected can be a far node (from secondary DC in case of AHA) and the upgrade can run for a long time. The upgrade job will download the WAR file and upgrade the node where this job is triggered. This node is then restarted and the second job "Check Upgrade Script/Check database for possible upgrade" will be triggered on System start-up and will continue the Upgrade. Both these jobs run on a worker thread. Modifying the Next action time of the "Upgrade/Check distribution for possible upgrade" job after it went past the "Planned start time" of the ServiceNow Support Change is not a recommended approach as this can create issues. Please plan your change "Planned start time" about 10-15 minutes before the run time of this job. Or you can adjust the Next action time (minutes), maybe 2-3 hours before the "Planned start time" of the change so that it is in lieu of the start of the change and it could run a few cycles successfully. 5. Which is the best process to DRY RUN my upgrade? Getting an ETA/behavior of a PROD upgrade is very important. To record the correct timelines/behavior there are 2 options: Initiate a FULL clone from PROD to SUBPROD including (Default is EXCLUDE) "Attachments", "Audit and log", and "Exclude tables specified in Exclusion list" (Uncheck all checkboxes for Exclusion so that they are INCLUDED) and then upgrade the SUB PROD instance. Including the above-specified tables are very important as these are the largest tables on an instance and index creation/schema changes to this table can take a major chunk of the total upgrade time. If they are excluded, that time cannot be factored into the planned upgrade activity. From London, during a clone, you have the option to select the amount of data on task tables( Default is 90 days). Please make sure you are selecting "Full" as we need the full task table to analyze the correct timelines.Request a restore of your PROD instance backup to the SUB PROD instance so that the SUB PROD instance can be an exact replica. SUB PROD can be upgraded and this should give you a definite ETA on the upgrade run time. Upgrades should be DRY RUN multiple times. If we do proper testing of the upgrade on SUB PROD the upgrade on PROD will go fine and the support is available to look into any unforeseen issue. Customers are encouraged to go to Now Support Help Center for available options. 6. For how long will my upgrade run? This is a broader question and we are unable to provide an ETA for this. Every instance is different from Organisational specific customizations and the upgrade time can vary based on the extent of data/customizations on the instance. DRY RUN is the only way to get this information. 7. What happens in my instance during an upgrade? During an upgrade, users can still be logged on to the instance and work as usual. There will be minimal impact and the user connection will be reset once when the node to which the user is connected is getting upgraded. Record creation and records created will not be impacted by the upgrade. Only the Scripts and the Schema will be modified via upgrade. One node will be running the whole process by upgrading the node and then triggering the upgrade of the Schema/DB. In parallel, it will run the upgrade on other nodes and after the upgrade, nodes will be restarted resetting the connections. So once the nodes are restarted, the nodes will be in a newer Version and the DB will not be in a newer version until the Database upgrade is completed. Even though record creations are not interrupted; my personal recommendation will be to restrict users during the upgrade. By this, the system can use all the available resources for the upgrade activity. 8. Can we take an ad-hoc backup prior to the upgrade? Unfortunately, there is no option to run an ad-hoc backup. We take separate backups for Primary and Secondary DB (where available). The backup cycle consists of weekly full backups and daily differential backups that provide 14-28 days(based on the retention policy) of backups. All backups are written to disk, no tapes are used and no backups are sent off-site. All the controls that apply to live customer data also apply to backups. If data is encrypted in the live database then it will also be encrypted in the backups." More information is available in the article Backup Request for ServiceNow instance including Production More information on Backups and Retention is covered in this white paper: Delivering Performance, Scalability, and Availability on The ServiceNow Cloud Data backup and recovery 9. Which are my options to recover my data in case of Data corruption after an upgrade? Your DATA is safe with us. In a most unlikely scenario of data corruption in PROD, we have the option to do a POINT IN TIME restore of the PROD instance to a TEMP instance by which we can bring back the data on the PROD instance to that specific time for data comparison (to a specific minute) on the TEMP instance. A point-in-time restore is done by restoring the instance to the last available backup and recreating the remaining data specified from the BIN logs/Transaction Logs. This process can take more time based on the amount of INSERT/DELETE/UPDATE ran on the instance from the last backup. Auto retention of BIN LOGS is 4 days as of this writing. PIT restore is a manual process involving multiple teams and this will be initiated in Critical conditions and cannot be treated as a Failover/Recovery/Back-out/Rollback Method. Note: We do not do PIT restore directly on PROD and we will only fix forward and the PIT restore will be used only for Data comparison and Data recovery on TEMP/SUBPROD instances. Once you have the data on the TEMP instance, this can be used to compare the data between POST and PRE upgrade versions and work on a data recovery plan. We can assist with any roadblocks as we have defined SOP to analyse and assist in recovering the data based on individual cases via the Change Management process. Fix Forward vs Restore from Backup Data Loss or Corruption on non - production/Sub Production instances KB explains different scenarios and Self Service automatons available for initiating a Restore. Please make sure to save your Development work before this. 10. What are the features impacted on the instance/application during an upgrade? Update-set preview/commit and plugin/app installations will not be available during the upgrade. It will display the message 'Info MessageUpdate set preview and commit are unavailable because the system is currently upgrading. Click here for the Upgrade Monitor'. This will resume after the upgrade is complete ( closure of upgrade change) See this article for more information about the impact: KB0622951 - Features impacted during the database upgrade If any of your integrations are not working during the upgrade, there is a high probability that the job is NOT listed as "Upgrade Safe". We recommend NOT modifying during the course of an upgrade as we cannot comment on the aftereffect. Custom scheduled jobs should NOT BE configured as "Upgrade Safe" as this field is there for a reason so that this will not interfere with the upgrade. 11. How to restrict users during an upgrade? There is no OOB option available to restrict users during an upgrade. There are a few workarounds and customization to achieve this and this should be implemented with your Developers/Partners and out of Customer support scope. These are a few available options: If your instance has Single Sign-On implemented, you can restrict access via the portal/identity provider for SSO/SAML.You can also run a custom script to log out all logged-in users prior to the change/upgrade window.You can also create a Local Admin account and only admin users can access the Instance with a Local Account of the instance skipping the SAML/SSO login 12. Can ServiceNow Tech Support monitor my upgrade? An upgrade is an automated process and we do not have any option to FastTrack upgrade once it is started. Any functional/operational issues encountered during an upgrade will have to wait until the upgrade is completed. We have seen most issues encountered during an upgrade will be rectified itself post-upgrade as there are a lot of moving parts during an upgrade where there will be new fields/functionality getting added and removed during the course. Monitoring the upgrade from the back-end localhost node log is a non-realistic approach as there will be up to 150 lines created in the back-end per second. The best way to Monitor the upgrade is via the Upgrade monitor window. Customer support also uses this window to monitor the upgrades because of the above-said reasons. See Upgrade Monitor Overview for more information. The Upgrade monitor will tell the number of plugins remaining and an estimate of what is upgrading and an estimate of remaining records per plugin as and when it runs. "sys_update" is a very important table where the duration of each activity is stored during an upgrade. So if you think your update/upgrade is stuck, you can look for this table on your SUB PROD DRY RUN ( only if it was a full clone of PROD) to get a ballpark estimate to compare with the PROD upgrade. In specific cases, Tech support monitors the upgrade for active issues raised previously which were tracked and rectified during the SUB PROD dry run and it would be communicated to the customer via an active case. We do not recommend creating placeholder cases for Tech Support to monitor upgrades for the above-said reason. We have different modules and SME teams for each functionality and a placeholder case will not help here as the cases should be addressed for individual teams for respective issues. In case of any issues observed during/after an upgrade, customers are encouraged to go to Now Support Help Center for assistance and we will be able to assist you with that specific issue. We have proactive monitoring in place for any instance/infra-specific resource constraint issues at all times. In case of any resource constraints automated alerts are created and we will be working with the customers on these issues as separate tickets to remediate this. Our monitoring solution reacts at the time of the event and takes appropriate action as needed which negates the need for Customer Support proactive monitoring. Were proactive monitoring to be conducted by support engineers on an instance, this would be periodically every few hours and could lead to missing an event, or not reacting fast enough due to external factors. For these reasons, ServiceNow Customer Support relies on our automated monitoring system. More information on ServiceNow Monitoring - Overview and Insight. 13. What to do when the node upgrade has failed/node is RED in Upgrade Summary? Nodes are just placeholder JVM for the users to access the DB. We do not have data on the nodes and if the nodes are down, there is no impact on the instance and there are automated monitoring and rules in place to sort this out automatically. All the user traffic will be diverted to the available nodes until this is sorted. If you notice any nodes are having issues/down/failed during the course of the upgrade, please wait till the Upgrade change is auto-closed. This will take up to 20-30 minutes from the Upgrade Summary report is generated. If the issue is not resolved, please call Customer Support and we will manually upgrade the node or spin up a new node on the new version and disable the stale one. 14. Can I roll back/Downgrade an upgrade? Unfortunately, there is NO option to run a rollback if the upgrade was to a different family. Only Patch upgrades can be rolled back. If customer support is initiating the Rollback on PROD via Change Management, we will have to test this on a SUB PROD to make sure we are getting the intended result set. For this, we will restore PROD to a SUB PROD and try the Rollback to check if it is getting the intended result-set and get customer confirmation, and then do it on PROD. Note: The rollback does not record schema drops (tables or columns, index drops are ok), re-parenting / column promotion, table truncate, table/column rename, column type changes, or column width decrease. These are excluded (exclusion list) from the rollback and are not configurable. ServiceNow recommends Fix Forward in case of any unforeseen catastrophic issues encountered after an upgrade/an accidental upgrade. Fix Forward:- Create individual cases for critical issues so that the respective SME team can work on this on Priority. Please mention the Upgrade change details on these cases. This KB is from a Data Loss context, but the scope is similar in this scenario as well as we recommend Fixing Forward. Fix Forward vs Restore from Backup You might also need to look at the section for more information about data recovery options: Which are my options to recover my data in case of Data corruption after an upgrade? More information on the above in the documentation: Rollback and delete recovery A rollback will create a property "glide.war.no_upgrade" with the value set to the version before the rollback. The presence of this property will ensure the instance is not upgraded to that version. From London, the instance administrator can trigger this: Rollback patch upgrades or plugin activations Upgrade Rollback Mechanism The downgrade of an instance is not an option for any instances. If the instance is a SUB PROD instance, it can be Cloned/Restored over from PROD/Instance which is of a lower version and that will make sure post-Clone/Restore automation, the target instance will match the same version of the Source instance. Data Loss or Corruption on non - production/Sub Production instances KB explains different scenarios and Self Service automatons available for initiating a Restore. Please make sure to save your Development work before this. 15. How to use an Update Set to capture reverted customizations made after an upgrade? The following article will help you with steps to capture the review of Skipped records on SUB PROD instance that can be reused after an upgrade on PROD: How to use an Update Set to capture reverted customizations made after an upgrade An update set will NOT capture the Disposition, Resolution, Comment, etc. from the Instance's upgrade log. Upon committing the update set on the new instance, these log entries( required columns) should be transferred using Import sets for housekeeping purposes. From Paris, this is changed and please find the KB to achieve this. Skipped Update Records Resolution Tracking - Upgrade related changes in Paris version From Tokyo+ this is changed and please find the KB where a lot more features are added and where you can automate a lot of Post upgrade activities Upgrade Plan module FAQ (Tokyo+) The "Upgrade Plan" module is not supported for organisations with multiple prod stacks linked to the same organisation, as these prod stacks use the same apprepo with multiple builder instances. 16. Why is the skipped record count different from my SUB PROD instance? We can only compare Apples to Apples. If the instances are not in sync; the number of records will be different between instances. We have seen this issue where the count is different between PROD and SUB PROD. This will happen if the SUB PROD was not a FULL clone (maybe the logs/audit were excluded) of PROD. Please check the section What is the best process to DRY RUN my upgrade? for more information. Please see KB0955553: Skipped Update Records Resolution Tracking - Upgrade related changes in Paris version 17. Why is there a skipped updates count mismatch between the Upgrade Summary and the Upgrade History records? Please see this article for more information: Skipped updates count mismatch between "Upgrade Summary Report in Upgrade Monitor" vs "Skipped updates widget on Upgrade History" vs "disposition Group by from sys_upgrade_history_log" table. The best place to get a consolidated list of what happened to the records during an upgrade is from the table sys_upgrade_history_log and Group by "Disposition" 18. Can we cancel the upgrade while it is in progress? An upgrade cannot be canceled gracefully once triggered on the instance. We have to let it run through the upgrade cycle and address any issues post-upgrade via ServiceNow Support tickets. Customers are encouraged to go to Now Support Help Center for available options. 19. How do I review Skipped Update Records and track its resolution? (new Paris & after feature) Please see KB0955553: Skipped Update Records Resolution Tracking - Upgrade related changes in Paris version 20. How to notify upcoming upgrade on your P+instance How to notify upcoming upgrade on your own instance. 21. Revert customized record to OOB and transfer via Update-set for Upgradeability Please check this KB Revert customized record to OOB and transfer via Update-set for Upgradeability 22. How to ensure workarounds applied are overwritten with fixed OOB records during the upgrade Please check this KB How to ensure workarounds applied are overwritten with fixed OOB records during the upgrade 23. Can I view an upgrade Summary Report for an older upgrade? Yes you can, if you click on the Upgrade monitor module it will always send you to the latest upgrade, but if you want to show the Summary Report for an older upgrade you can open the upgrade history module, open the upgrade record, and on the form will be a Related link: View Upgrade Summary Report. This will open up the summary report for that given upgrade.