MID Server Upgrade Fails, after manually copying more ZIP files than necessary into agent\package\incomingIssue This specific issue can happen only if you used this technique to upgrade MID Servers:KB0565184 - How to upgrade a MID Server that does not have access to the AutoUpgrade install server on the Internet The MID Server will report "Upgrade Failed" but remain Up. The agent log will show errors "bootstrapping upgrade", which means it didn't successfully run the command to start the separate upgrade process, using the files extracted to the temp folder. 12/21/22 07:57:46 (376) AutoUpgrade.3600 Stopping MID server. Bootstrapping upgrade.12/21/22 07:57:46 (376) AutoUpgrade.3600 MIDUpgrader.startUpgradeRunner(), OperationalState=UPGRADING...12/21/22 07:59:02 (053) AutoUpgrade.3600 SEVERE *** ERROR *** Upgrade failed.com.service_now.mid.dist.upgrade.UpgradeException: Failed to bootstrap. at com.service_now.mid.dist.upgrade.MIDUpgrader.startUpgradeRunner(MIDUpgrader.java:413) at com.service_now.mid.dist.upgrade.MIDUpgrader.upgrade(MIDUpgrader.java:264) at com.service_now.monitor.AutoUpgrade.attemptUpgrade(AutoUpgrade.java:241) at com.service_now.monitor.AutoUpgrade.run(AutoUpgrade.java:119) at com.service_now.monitor.AMonitor.runit(AMonitor.java:155) at com.service_now.monitor.AMonitor.access$200(AMonitor.java:44) at com.service_now.monitor.AMonitor$MonitorTask.execute(AMonitor.java:148) at com.service_now.monitor.base.AMonitorTask.run(AMonitorTask.java:29) at java.base/java.util.TimerThread.mainLoop(Timer.java:556) at java.base/java.util.TimerThread.run(Timer.java:506) The wrapper log for the upgrade process showing the Tanuki Wrapper error Failed to resolve the version of Java, can be found in the temp folder:agent/work/upgrade_temp/<some number>/upgrade-wrapper/logs/glide-dist-upgrade.log STATUS | wrapper | 2022/12/21 07:58:58.499 | --> Wrapper Started as DaemonSTATUS | wrapper | 2022/12/21 07:58:58.500 | Java Service Wrapper Standard Edition 64-bit 3.5.40STATUS | wrapper | 2022/12/21 07:58:58.500 | Copyright (C) 1999-2019 Tanuki Software, Ltd. All Rights Reserved.STATUS | wrapper | 2022/12/21 07:58:58.500 | http://wrapper.tanukisoftware.comSTATUS | wrapper | 2022/12/21 07:58:58.500 | Licensed to ServiceNow, Inc. for MID UpgradeSTATUS | wrapper | 2022/12/21 07:58:58.500 | FATAL | wrapper | 2022/12/21 07:58:59.376 | Failed to resolve the version of Java.STATUS | wrapper | 2022/12/21 07:58:59.579 | <-- Wrapper Stopped The script that is actually being run to start this java upgrade process is:agent/work/upgrade_temp/<some number>/upgrade-wrapper/bin/glide-dist-upgrade.sh start It launches a child process for the Tanuki Wrapper. The first thing that does is run "java -version" to check the java version is compatible with the Tanuki Wrapper. If you manually run "java -version" from the extracted JRE in the temp folder, you will see the real Java error: xxx@yyy:.../agent/work/upgrade_temp/13408208836640019553/upgrade-wrapper-jre/bin> ./java -versionError occurred during initialization of VMjava.lang.UnsatisfiedLinkError: 'long sun.io.Win32ErrorMode.setErrorMode(long)'at sun.io.Win32ErrorMode.setErrorMode(java.base/Native Method)at sun.io.Win32ErrorMode.initialize(java.base/Win32ErrorMode.java:72)at jdk.internal.misc.OSEnvironment.initialize(java.base/OSEnvironment.java:39)at jdk.internal.misc.VM.initializeOSEnvironment(java.base/VM.java:236)at java.lang.System.initPhase1(java.base/System.java:1984)CauseThe cause is copying both the Windows and Linux JRE ZIP files into the package folder. (or x86 and x86 ZIP files) The agent log will show that both were extracted to the same place. This results in both Linux "java" and Windows "java.exe" being in the upgrade_temp folders, but all the dependent fiels will either be for Linux or Windows. The mismatch in OS/architecture between the java executable and the dependent files causes the error when the wrapper attempts to run "java -version" to get the java version. 12/21/22 07:57:38 (772) AutoUpgrade.3600 Current packages:12/21/22 07:57:38 (772) AutoUpgrade.3600 Installed: [mid-core.sandiego-12-22-2021__patch8-09-14-2022_09-27-2022_1616.universal.universal.zip, mid-jre.sandiego-12-22-2021__patch3-04-27-2022_05-11-2022_1201.linux.x86-64.zip]12/21/22 07:57:38 (772) AutoUpgrade.3600 Assigned: [mid-jre.tokyo-07-08-2022__patch4-11-23-2022_12-05-2022_2132.linux.x86-64.zip, mid-core.tokyo-07-08-2022__patch4-11-23-2022_12-05-2022_2132.universal.universal.zip, mid-upgrade.tokyo-07-08-2022__patch4-11-23-2022_12-05-2022_2132.universal.universal.zip]12/21/22 07:57:38 (772) AutoUpgrade.3600 Missing: [mid-jre.tokyo-07-08-2022__patch4-11-23-2022_12-05-2022_2132.linux.x86-64.zip, mid-core.tokyo-07-08-2022__patch4-11-23-2022_12-05-2022_2132.universal.universal.zip, mid-upgrade.tokyo-07-08-2022__patch4-11-23-2022_12-05-2022_2132.universal.universal.zip]12/21/22 07:57:38 (772) AutoUpgrade.3600 Downloaded: [mid-core.tokyo-07-08-2022__patch4-11-23-2022_12-05-2022_2132.universal.universal.zip, mid-jre.tokyo-07-08-2022__patch4-11-23-2022_12-05-2022_2132.linux.x86-64.zip, mid-jre.tokyo-07-08-2022__patch4-11-23-2022_12-05-2022_2132.windows.x86-64.zip, mid-upgrade.tokyo-07-08-2022__patch4-11-23-2022_12-05-2022_2132.universal.universal.zip]...12/21/22 07:57:39 (991) AutoUpgrade.3600 Extracting package .../agent/package/incoming/mid-core.tokyo-07-08-2022__patch4-11-23-2022_12-05-2022_2132.universal.universal.zip to .../agent/work/upgrade_temp/829104795661780239712/21/22 07:57:43 (121) AutoUpgrade.3600 Extracting package .../agent/package/incoming/mid-jre.tokyo-07-08-2022__patch4-11-23-2022_12-05-2022_2132.linux.x86-64.zip to .../agent/work/upgrade_temp/829104795661780239712/21/22 07:57:43 (892) AutoUpgrade.3600 Extracting package .../agent/package/incoming/mid-jre.tokyo-07-08-2022__patch4-11-23-2022_12-05-2022_2132.windows.x86-64.zip to .../agent/work/upgrade_temp/829104795661780239712/21/22 07:57:44 (904) AutoUpgrade.3600 Extracting package .../agent/package/incoming/mid-upgrade.tokyo-07-08-2022__patch4-11-23-2022_12-05-2022_2132.universal.universal.zip to .../agent/work/upgrade_temp/8291047956617802397 This is a Linux server. mid-jre for linux is extracted, then the wrong files from windows are extracted over the top.ResolutionDelete any ZIP files from the agent\package\incoming folder that are for the wrong OS or architecture, then restart the mid server service. The upgrade will then work. Next time, only copy the ZIP files that are actually needed by the MID Server.