Since Washington DC, MID Server upgrades can fail due to a file lock on wrapper.conf, leaving the MID Server down and unable to start without manual file repair - Known Error

Since Washington DC, MID Server upgrades can fail due to a file lock on wrapper.conf, leaving the MID Server down and unable to start without manual file repair Description Since Washington DC, MID Server upgrades can fail due to a file lock on wrapper.conf, leaving the MID Server down and unable to start without manual file repair.

This problem is tracking only failed MID Server upgrades, on Windows, since WashingtonDC, where the glide-dist-upgrade.log contains:

SEVERE: com.snc.dist.mid_upgrade.UpgradeException: java.nio.file.FileSystemException: ...\agent\conf\wrapper.conf: The process cannot access the file because it is being used by another process

Steps to Reproduce

The cause is currently unknown, but is happening since Washington.

NOTE: ServiceNow support will need to learn more about customer server environments to understand what is locking the file. The assumption is some kind of anti-virus software, but exactly what, and why is not known. And support cases linking this problem will need to capture all logs and tasklist, and what anti-virus is running

What happens is: The agent log shows the MID realises it has to upgrade, and starts the process

2024-06-08T08:02:36.451+0200 INFO (AutoUpgrade.3600.now) [Packages:148] Missing: [mid-jre.washingtondc-12-20-2023__patch1-hotfix2a-04-02-2024_04-05-2024_1127.windows.x86-64.zip, mid-core.washing... 2024-06-08T08:02:36.623+0200 INFO (AutoUpgrade.3600.now) [Instance:918] Setting mid status to Upgrading During the pre-upgrade checks it may have some issues, that are not blocking. Not all customers see these.

2024-06-08T08:02:39.467+0200 INFO (AutoUpgrade.3600.now) [UserPermissions:115] Start/stop permissions check 2024-06-08T08:02:39.545+0200 ERROR (AutoUpgrade.3600.now) [UserPermissions:54] Unable to fetch sid of user: xxx com.service_now.mid.util.UserPermissionsException: Unable to fetch sid of user: xxx 2024-06-08T08:07:44.828+0200 WARN (AutoUpgrade.3600.now) [APowershellSession:366] Command: 81a6e8f4-a3d8-47a1-b734-caa6f2b01df7 with id: snc-decode-command cnVuVXBncmFkZUNoZWNrICJzbmNfbWlkXzIwMjIi | invoke-expression timed out after 300 seconds:, command: runUpgradeCheck "snc_mid_2022", stdout: [], stderr[] com.snc.automation_common.integration.exceptions.CommandTimeoutException: Command: 81a6e8f4-a3d8-47a1-b734-caa6f2b01df7 with id: snc-decode-command cnVuVXBncmFkZUNoZWNrICJzbmNfbWlkXzIwMjIi | invoke-expression timed out after 300 seconds 2024-06-08T08:09:45.029+0200 WARN (AutoUpgrade.3600.now) [PowerConsole:354] PowerConsole unable to close 2024-05-18T12:34:10.313-0400 INFO (AutoUpgrade.3600) [UserPermissions:161] Executing command `sc sdshow ServiceNow Mid ITSM Prod`, Working dir: `C:\ServiceNow\Prod\ServiceNow MID Server ServiceNow ITSM Prod\agent`, Current user: `xxxxx` 2024-05-18T12:34:10.466-0400 INFO (AutoUpgrade.3600) [UserPermissions:163] Execution result: ExecuteException: Process exited with an error: 1060 (Exit value: 1060) 2024-05-18T12:34:10.466-0400 ERROR (AutoUpgrade.3600) [UserPermissions:55] Failed executing command : sc sdshow ServiceNow Mid ITSM Prod com.service_now.mid.util.UserPermissionsException: Failed executing command : sc sdshow ServiceNow Mid ITSM Prod at com.service_now.mid.util.UserPermissions.hasStartStopPermissions(UserPermissions.java:129) at com.service_now.mid.util.UserPermissions.validate(UserPermissions.java:53) at com.service_now.mid.dist.upgrade.MIDUpgrader.runPreUpgradeCheck(MIDUpgrader.java:579) at com.service_now.monitor.AutoUpgrade.attemptUpgrade(AutoUpgrade.java:176) at com.service_now.monitor.AutoUpgrade.run(AutoUpgrade.java:121) at com.snc.midserver.monitor.internal.MonitorRunner$MonitorTask.execute(MonitorRunner.java:234) at com.snc.midserver.monitor.internal.AMonitorTask.run(AMonitorTask.java:29) at java.base/java.util.TimerThread.mainLoop(Timer.java:556) at java.base/java.util.TimerThread.run(Timer.java:506) An upgrade system command may be seen while the pre-upgrade check runs:

2024-06-08T08:02:44.674+0200 INFO (Thread-209493) [AWorker:136] Worker completed: SystemCommand source: restart time: 0:00:00.094 But tests are successful and it goes ahead:

2024-06-08T08:09:47.355+0200 INFO (AutoUpgrade.3600.now) [MIDUpgrader:601] Pre-upgrade validation tests successful. Continuing with upgrade process. 2024-06-08T08:12:31.113+0200 INFO (AutoUpgrade.3600.now) [MIDUpgrader:501] Stopping MID server. Bootstrapping upgrade. And it does get the confirmation from the upgrade process that it was bootstrapped

2024-06-08T08:12:22.629+0200 INFO (AutoUpgrade.3600.now) [MIDUpgrader:1015] Writing MID and wrapper PIDs to D:\MidSrv2022\ServiceNow MID Server Mid_Server_2022\agent\conf\pids 2024-06-08T08:12:45.408+0200 INFO (AutoUpgrade.3600.now) [MIDUpgrader:375] DistUpgrade file is present.that it has started 2024-06-08T08:12:45.424+0200 INFO (AutoUpgrade.3600.now) [MIDUpgrader:317] Upgrade process start successful and so shuts down 2024-06-08T08:12:49.818+0200 INFO (MIDServer) [Bootstrapper:60] the ServiceNow MID Server is now terminated wrapper log confirms that:

2024/06/08 08:12:52 | meanwhile the upgrade process hits a file lock for the agent\conf\wrapper.conf , and stops after it has deleted stuff, but before it has copied everything back

INFO | jvm 1 | 2024/06/08 08:12:39.809 | Jun 08, 2024 8:12:39 A.M. com.snc.dist.mid_upgrade.DistUpgradeUtil writeStartedStatusToFile INFO | jvm 1 | 2024/06/08 08:12:52.575 | INFO: Executing command `sc query "snc_mid_2022"`. Working dir: `D:\MidSrv2022\ServiceNow MID Server Mid_Server_2022\agent`. Current user: `SVC-SN-EUWS2870`. INFO | jvm 1 | 2024/06/08 08:12:52.684 | Jun 08, 2024 8:12:52 A.M. com.snc.dist.mid_upgrade.UpgradeMain executeCommand INFO | jvm 1 | 2024/06/08 08:12:52.684 | INFO: Execution result: ExitCode=0 INFO | jvm 1 | 2024/06/08 08:12:52.684 | Output= INFO | jvm 1 | 2024/06/08 08:12:52.684 | SERVICE_NAME: snc_mid_2022 INFO | jvm 1 | 2024/06/08 08:12:52.684 | TYPE : 10 WIN32_OWN_PROCESS INFO | jvm 1 | 2024/06/08 08:12:52.684 | STATE : 1 STOPPED INFO | jvm 1 | 2024/06/08 08:12:52.684 | WIN32_EXIT_CODE : 0 (0x0) INFO | jvm 1 | 2024/06/08 08:12:52.684 | SERVICE_EXIT_CODE : 0 (0x0) INFO | jvm 1 | 2024/06/08 08:12:52.684 | CHECKPOINT : 0x0 INFO | jvm 1 | 2024/06/08 08:12:52.684 | WAIT_HINT : 0x0

INFO | jvm 1 | 2024/06/08 08:12:52.913 | Jun 08, 2024 8:12:52 A.M. com.snc.dist.mid_upgrade.WindowsMIDUpgradeUtil isPidRunning INFO | jvm 1 | 2024/06/08 08:12:52.913 | INFO: Process (pid=6124) is not running. INFO | jvm 1 | 2024/06/08 08:12:53.132 | Jun 08, 2024 8:12:53 A.M. com.snc.dist.mid_upgrade.WindowsMIDUpgradeUtil isPidRunning INFO | jvm 1 | 2024/06/08 08:12:53.132 | INFO: Process (pid=3024) is not running.

INFO | jvm 1 | 2024/06/08 08:13:51.022 | Jun 08, 2024 8:13:50 A.M. com.snc.dist.mid_upgrade.UpgradeMain run INFO | jvm 1 | 2024/06/08 08:13:51.022 | SEVERE: com.snc.dist.mid_upgrade.UpgradeException: java.nio.file.FileSystemException: D:\MidSrv2022\ServiceNow MID Server Mid_Server_2022\agent\conf\wrapper.conf: The process cannot access the file because it is being used by another process INFO | jvm 1 | 2024/06/08 08:13:51.022 | com.snc.dist.mid_upgrade.UpgradeException: java.nio.file.FileSystemException: D:\MidSrv2022\ServiceNow MID Server Mid_Server_2022\agent\conf\wrapper.conf: The process cannot access the file because it is being used by another process INFO | jvm 1 | 2024/06/08 08:13:51.022 | at com.snc.dist.mid_upgrade.UpgradeMain.migrateToTarget(UpgradeMain.java:1051) INFO | jvm 1 | 2024/06/08 08:13:51.022 | at com.snc.dist.mid_upgrade.UpgradeMain.run(UpgradeMain.java:365) INFO | jvm 1 | 2024/06/08 08:13:51.022 | at java.base/java.lang.Thread.run(Thread.java:833) INFO | jvm 1 | 2024/06/08 08:13:51.022 | Caused by: java.nio.file.FileSystemException: D:\MidSrv2022\ServiceNow MID Server Mid_Server_2022\agent\conf\wrapper.conf: The process cannot access the file because it is being used by another process INFO | jvm 1 | 2024/06/08 08:13:51.022 | at java.base/sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:92) In this situation the upgrade process stops, and does not risk starting the mid server service again.

No other wrapper logs are seen after the main service shutdown. (As Washington doesn't use "mid.bat status" any more, you don't even see those running:no outputs.)

When the MID is manually started, it can't, probably due to missing files due to the incomplete migrateToTarget:

2024-06-08T09:38:53.541+0200 INFO (WrapperStartStopAppMain) [ALoggerConfigProvider:56] Logger config: root=INFO 2024-06-08T09:38:58.449+0200 ERROR (WrapperStartStopAppMain) [Bootstrapper:57] terminating due to fatal error

Workaround There is no workaround, or known way to prevent this, as the cause of this problem is not yet know. Please tell ServiceNow support about this, provide the logs, and deatails about the host including any monitoring agents and anti-virus software running.

Manually re-starting the upgrade process from the temp folder may work, although this has not worked in all cases KB0779816 How to continue a MID Server upgrade after it has crashed in the middle of the ServiceNow Platform Distribution Upgrade service leaving the MID Server down and service not running In summary:

locate the upgrade_temp folder. e.g. c:\ServiceNow MID Server...\...\agent\work\upgrade_temp\ \ open a command prompt, as administrator and change directory to this sub-folder of the temp folder e.g. cd \upgrade-wrapper\bin\ run: wrapper-windows-x86-64.exe -c ..\conf\wrapper.conf If this isn't possible, a manual repair of the files of the MID Server is usually going to be possible, using this process: KB0713557 How to manually restore or upgrade a MID Server after a failed auto-upgrade

If all else fails, a new install might be the best and quickest option, although features will need updating to use the new MID Server instead.