MID Server Down after auto-Upgrade service 10 minute file lock timeout: SEVERE: java.io.IOException: Unable to delete agent\lib\accessors-smart.jar - Known Error

MID Server Down after auto-Upgrade service 10 minute file lock timeout: SEVERE: java.io.IOException: Unable to delete agent\lib\accessors-smart.jar Description MID Server remains Down after the auto-Upgrade service times out after 10 minutes, while still waiting for agent\lib\accessors-smart.jar file lock to be freed up. The file remains locked by java.exe, which is the main MID Server service, that has not completely shut down yet.

The upgrade service does not finish, and does not start the MID Server again.

This problem applies only to Madrid and later MID Servers where the timeout is 10m, rather than 2m in earlier versions.

Steps to Reproduce

Specific real-world conditions required to reproduce unknown at this time. This can be forced by deliberately locking a file in lib.

glide-dist-upgrade.log within the temp folder shows:

ERROR | wrapper | 2019/11/04 10:48:22.961 | The ServiceNow Platform Distribution Upgrade ( ) service is not installed - The specified service does not exist as an installed service. (0x424) STATUS | wrapper | 2019/11/04 10:48:25.195 | ServiceNow Platform Distribution Upgrade ( ) service installed. STATUS | wrapper | 2019/11/04 10:48:32.071 | Starting the ServiceNow Platform Distribution Upgrade ( ) service... STATUS | wrapper | 2019/11/04 10:48:32.102 | --> Wrapper Started as Service STATUS | wrapper | 2019/11/04 10:48:32.134 | Java Service Wrapper Standard Edition 64-bit 3.5.17 STATUS | wrapper | 2019/11/04 10:48:32.149 | Copyright (C) 1999-2012 Tanuki Software, Ltd. All Rights Reserved. STATUS | wrapper | 2019/11/04 10:48:32.165 | http://wrapper.tanukisoftware.com STATUS | wrapper | 2019/11/04 10:48:32.180 | Licensed to ServiceNow, Inc. for MID Upgrade STATUS | wrapper | 2019/11/04 10:48:32.215 | STATUS | wrapper | 2019/11/04 10:48:32.587 | Launching a JVM... INFO | jvm 1 | 2019/11/04 10:48:32.821 | WrapperManager: Initializing... INFO | jvm 1 | 2019/11/04 10:48:33.055 | nov 04, 2019 10:48:32 AM com.snc.dist.mid_upgrade.UpgradeMain$1 start INFO | jvm 1 | 2019/11/04 10:48:33.087 | INFO: WrapperListener.start() INFO | jvm 1 | 2019/11/04 10:48:33.149 | nov 04, 2019 10:48:32 AM com.snc.dist.mid_upgrade.UpgradeMain logArguments INFO | jvm 1 | 2019/11/04 10:48:33.244 | INFO: Command-line arguments: --dist-file=C:\Windows\Temp\1572860822078-0\mid-upgrade.madrid-12-18-2018__patch7a-10-01-2019_10-03-2019_1530.universal.universal.zip --mode=dist-upgrade --current-dir= \agent INFO | jvm 1 | 2019/11/04 10:48:33.384 | nov 04, 2019 10:48:32 AM com.snc.dist.mid_upgrade.UpgradeMain run INFO | jvm 1 | 2019/11/04 10:48:33.434 | INFO: MID Server upgrade v2.0 starting... INFO | jvm 1 | 2019/11/04 10:48:33.446 | nov 04, 2019 10:48:32 AM com.snc.dist.mid_upgrade.UpgradeMain run INFO | jvm 1 | 2019/11/04 10:48:33.462 | INFO: Upgrading MID ` \agent` with contents from `C:\Windows\Temp\1572860822078-0\agent`. INFO | jvm 1 | 2019/11/04 10:48:33.477 | nov 04, 2019 10:48:33 AM com.snc.dist.mid_upgrade.UpgradeMain missingOrCanDelete INFO | jvm 1 | 2019/11/04 10:48:33.494 | INFO: \agent\lib\sigar-amd64-winnt.dll cannot be deleted: \agent\lib\sigar-amd64-winnt.dll (The process cannot access the file because it is being used by another process) INFO | jvm 1 | 2019/11/04 10:48:33.524 | Retrying in 1000ms... INFO | jvm 1 | 2019/11/04 10:48:34.087 | nov 04, 2019 10:48:34 AM com.snc.dist.mid_upgrade.UpgradeMain missingOrCanDelete INFO | jvm 1 | 2019/11/04 10:48:34.087 | INFO: \agent\lib\sigar-amd64-winnt.dll cannot be deleted: \agent\lib\sigar-amd64-winnt.dll (The process cannot access the file because it is being used by another process) INFO | jvm 1 | 2019/11/04 10:48:34.118 | Retrying in 1000ms... STATUS | wrapper | 2019/11/04 10:48:35.134 | ServiceNow Platform Distribution Upgrade ( ) started. INFO | jvm 1 | 2019/11/04 10:48:35.118 | nov 04, 2019 10:48:35 AM com.snc.dist.mid_upgrade.UpgradeMain missingOrCanDelete INFO | jvm 1 | 2019/11/04 10:48:35.149 | INFO: \agent\lib\sigar-amd64-winnt.dll cannot be deleted: \agent\lib\sigar-amd64-winnt.dll (The process cannot access the file because it is being used by another process) ... INFO | jvm 1 | 2019/11/04 10:48:47.213 | nov 04, 2019 10:48:47 AM com.snc.dist.mid_upgrade.UpgradeMain missingOrCanDelete INFO | jvm 1 | 2019/11/04 10:48:47.213 | INFO: \agent\lib\sigar-amd64-winnt.dll cannot be deleted: \agent\lib\sigar-amd64-winnt.dll (The process cannot access the file because it is being used by another process) INFO | jvm 1 | 2019/11/04 10:48:47.244 | Retrying in 1000ms... INFO | jvm 1 | 2019/11/04 10:48:48.229 | nov 04, 2019 10:48:48 AM com.snc.dist.mid_upgrade.UpgradeMain missingOrCanDelete INFO | jvm 1 | 2019/11/04 10:48:48.244 | INFO: \agent\lib\sigar-amd64-winnt.dll is no longer locked after 15 checks. INFO | jvm 1 | 2019/11/04 10:48:48.244 | nov 04, 2019 10:48:48 AM com.snc.dist.mid_upgrade.UpgradeMarker markService INFO | jvm 1 | 2019/11/04 10:48:48.260 | INFO: Added service `snc-platform-dist-upgrade- ` to upgrade marker file. INFO | jvm 1 | 2019/11/04 10:58:51.863 | nov 04, 2019 10:58:51 AM com.snc.dist.mid_upgrade.UpgradeMain timedDelete INFO | jvm 1 | 2019/11/04 10:58:51.863 | SEVERE: Timeout deleting \agent\lib\accessors-smart.jar INFO | jvm 1 | 2019/11/04 10:58:51.879 | nov 04, 2019 10:58:51 AM com.snc.dist.mid_upgrade.UpgradeMain run INFO | jvm 1 | 2019/11/04 10:58:51.879 | SEVERE: java.io.IOException: Unable to delete ` \agent\lib\accessors-smart.jar`, MID Server upgrade cannot continue. If the MID Server will not restart you may need to reinstall the MID server. INFO | jvm 1 | 2019/11/04 10:58:51.894 | java.io.IOException: Unable to delete ` \agent\lib\accessors-smart.jar`, MID Server upgrade cannot continue. If the MID Server will not restart you may need to reinstall the MID server. INFO | jvm 1 | 2019/11/04 10:58:51.925 | at com.snc.dist.mid_upgrade.UpgradeMain.wipeDirectory(UpgradeMain.java:450) INFO | jvm 1 | 2019/11/04 10:58:51.925 | at com.snc.dist.mid_upgrade.UpgradeMain.wipeDirs(UpgradeMain.java:398) INFO | jvm 1 | 2019/11/04 10:58:51.941 | at com.snc.dist.mid_upgrade.UpgradeMain.run(UpgradeMain.java:309) INFO | jvm 1 | 2019/11/04 10:58:51.941 | at java.lang.Thread.run(Thread.java:748) INFO | jvm 1 | 2019/11/04 10:58:51.957 | INFO | jvm 1 | 2019/11/04 10:58:51.957 | nov 04, 2019 10:58:51 AM com.snc.dist.mid_upgrade.UpgradeMain appendMidLogs INFO | jvm 1 | 2019/11/04 10:58:51.972 | INFO: Flushing logs Workaround This problem is currently under review. You can contact ServiceNow Technical Support or subscribe to this Known Error article by clicking the Subscribe button at the top right of this form to be notified when more information will become available..

To get the MID Server working again after this has happened:

Although the error suggests "If the MID Server will not restart you may need to reinstall the MID server", that is usually not the best approach. Don't try to start the main MID Server service. Instead try this, and the upgrade will probably successfully complete:

Check the main MID Server service is now stopped, and kill the java.exe process if necessary. Follow the steps in KB0779816 How to continue a MID Server upgrade after it has crashed in the middle of the ServiceNow Platform Distribution Upgrade service, leaving the MID Server Down and the Service not running There is no workaround to avoid this issue happening at present.