MID Server upgrade process - What actually happens when a MID Server upgrades itself?Issue How does the MID server upgrade process work? Knowing this will help you debug if it goes wrong, and identify exactly where it went wrong. This article describes the process that takes place when the MID server auto-upgrades, which should happen immediately after an instance upgrade of a patch finishes. This is based on a Windows host and Vancouver version. The Linux process is basically the same, but with different temp folder locations and using shell scripts instead of batch files. MID Server Upgrade Process on a Windows host The Upgrade check is triggered in one of these ways: By the StartupSequencer thread when the MID Server starts up.At the end of an Instance patch or upgrade, when the instance sends a topic=SystemCommand, source=autoUpgrade job to all MID Servers that are UP at the time via the ECC Queue. Other MID Server related plugins may also send Restart requests to the MID Server at this time.1.1 Every hour, when the MID Server's "AutoUpgrade.3600" thread runs. You cannot stop that check happening.If the "Upgrade MID" related link is clicked on the MID Server form. "Checking to see if MID server needs to upgrade." will be written to the MID Server agent log, and the instance is queried to find out what version the MID Server should be: The MID Server knows the buildstamp of the 'core' and 'jre' packages installed from the contents of the agent\package\meta\mid-core.meta.properties and mid-jre.meta.properties files. 'core' is the MID Server application and is upgraded every time a MID Server Upgrades. The 'jre' is only likely to be upgraded every major relase so is likely to be an older buildstamp than the core.2.3It asks the instance what is Assigned using the "MIDAssignedPackages" Scripted SOAP Service. The request includes os, architecture and JVM version, and the corresponding core, jre and upgrade package names/URLs are returned: The assigned packages are derived from the MID Buildstamp of the instance version in the mid.buildstamp system property. That should always match the current instance version and be updated automatically with each instance upgrade. The instance stats.do page will also report the MID Buildstamp for manual confirmation.glide.war and glide.war.assigned system properties are used to see if a instance rollback has occurred.2.1If the specific instance app node that the SOAP request is handled by has not upgraded and restarted yet for some reason, the pre-upgrade version may still be returned2.2A MID Server's mid.pinned.version parameter will override the instance version. This is the only method that will prevent a MID Server upgrade when an instance upgrades, and is planned to be deprecated and removed from the documentation, due to the dangers of running a mismatched MID Server.2.4The assigned Java JRE version is hard-coded as a variable in the script of the SOAP web service. If this already matches the JRE installed in the MID Server, then the JRE upgrade will be skipped. This means that record must be the out-of-box version from the recent instance upgrade, and reverted if necessary. This script should never be customised or it will cause it to be skipped in upgrades.The Rome JRE upgrade will also be skipped if the Linux host is 32bit or if glibc version is <v2.17. 'Installed' is compared to 'Assigned', with the 'Missing' packages being the difference, and what will later be downloaded and installed in step 5.If the assigned version is older than the installed version, then a downgrade is attempted, however, in some cases this will cause problems. The older instance version may not have the newer code/APIs that the MID Server was expecting when starting the downgrade process, while the MID Server is still running future code. The MID Server agent log will report something like this. 'Assigned' will always include the mid-upgrade as well as the mid-core package. 'Assigned' and 'Missing' will include a mid-jre entry only if that needs to be upgraded, and in this example it didn't. An upgrade from Paris to Quebec, or Quebec to Rome, would include a JRE upgrade, but usually not for subsequent patches and hotfixes within the major version. Current packages: Installed: [mid-core.quebec-12-09-2020__patch0-hotfix3-01-20-2021_01-21-2021_0905.universal.universal.zip, mid-jre.quebec-12-09-2020__patch0-hotfix1-01-04-2021_01-06-2021_1339.windows.x86-64.zip] Assigned: [mid-upgrade.quebec-12-09-2020__patch1-02-18-2021_03-01-2021_1225.universal.universal.zip, mid-core.quebec-12-09-2020__patch1-02-18-2021_03-01-2021_1225.universal.universal.zip] Missing: [mid-upgrade.quebec-12-09-2020__patch1-02-18-2021_03-01-2021_1225.universal.universal.zip, mid-core.quebec-12-09-2020__patch1-02-18-2021_03-01-2021_1225.universal.universal.zip] Downloaded: [] "Setting mid status to Upgrading" will be written to the MID Server agent log The MID Server record will be set as Status=Upgrading3.1, and will be paused, and not take any more new jobs (except system commands).The MID Server Upgrade History records in the instance will be updated. "Performing pre-upgrade validation tests" will be written to the MID Server agent log These pre-checks will run by default, unless MID Server configuration parameter mid.upgrade.run_precheck=false. In that case Upgrade History "Pre Upgrade Check" Stage is marked as Skipped in the instance.Upgrade History "Pre Upgrade Check" Stage is marked as In Progress in the instance.For on-premise instances, isolated environments, or Regulated Market datacenter instances, it may not be possible to pass this test, requiring a workaround.4.1,4.8A "mid-upgrade...preUpgradeCheck.zip" file is downloaded from https://install.service-now.comSince Quebec, a Certificate check for host install.service-now.com is done. This should be fine as the certificate is usually the same as the connection to the instance.If the signed preUpgradeCheck.zip file fails Verifying digital signature, the upgrade will fail.4.4The contents are extracted to the TEMP folder. Since Rome, this is within agent\work. Quebec and earlier is the OS/Service User temp folder.In Windows, this includes a test where a simple PowerShell script is run to check the PowerShell version and user permissions.4.2,4.3,4.7Some known configurations that cause upgrades to fail are also checked, including that Application Experience is running4.6.The temp files are deleted.If all is well then "Pre-upgrade validation tests successful. Continuing with upgrade process". Upgrade History "Pre Upgrade Check" Stage is marked as Complete in the instance.If there are failures, specific errors or non-blocking warnings will be added to the agent log and to the MID Server Issue table [ecc_agent_issue]. 4.5. Upgrade History "Pre Upgrade Check" Stage is marked as Failed in the instance. Download the missing package ZIP files Upgrade History "Download Upgrade Packages" Stage is marked as In Progress in the instance.mid-upgrade...zip then mid-core...zip will always need downloading, and possibly also mid-jre...zip, if the Java Runtime also needs upgrading. The specific filenames needed are listed under the "Missing: [...]" line of the Current Packages check above.ZIP files are saved in the \agent\package\incoming\ folder.If the ZIP file contains a META-INF folder, Signatures are checked to make sure the ZIP file is not tampered with.4.5If all were logged as "Package was successfully downloaded" then we continue.If instance system property mid.download.through.instance=true, then ZIP files will be downloaded via the instance, and not directly from install.service-now.com. That should now be set false by default due to causing blocked semaphores in the instance. 5.1If the file is incomplete, perhaps due to socket timeout, the file is deleted and download retried. If maximum reties is reached, or there is a problem deleting the file, this needs resolving manually.Upgrade History "Download Upgrade Packages" Stage is marked as Complete or Failed in the instance. Extracting the ZIP files Upgrade History "Extract Upgrade Packages" Stage is marked as In Progress in the instance."Upgrading MID server" will be written to the MID Server agent log, now that we have everything we need.Extracts the ZIP files to the temp folder. Anti-virus/security software can block creating those temp files, breaking the upgrade6.5.Prior to Rome, including an upgrade to Rome or later from an earlier version, the OS temp folder is used, unless overwritten in wrapper-override.conf6.1. This is a random folder name like C:\Windows\TEMP\<random 13 digit number>-0\. Since then , a temporary folder under \agent\work\upgrade_temp\<random 13 digit number>-0\ is used. We have full control of permissions in that folder, and so will have fewer problems that we had with the OS shared temp folder. This would be a default behavior for any upgrade from Rome, once the instance has already been upgraded to Rome or later. mid.upgrade.use_os_temp_folder controls that behaviour, and defaults to false.Each file is logged with "Extracting:" separatelyUpgrade History "Extract Upgrade Packages" Stage is marked as Complete in the instance. Start the upgrade Process, and shut down Upgrade History "FipsConversion" Stage will usually be Skipped, with message "MID Server is not in FIPS Enforced mode or mid-jre is not being updated. No conversion required.". If FIPS approved only mode is enabled, the incoming JRE will contain a TrustStore (cacerts) file which may not be of the correct type (as of 3/30/2021 that type is BCFKS). Conversion of that file and updating the java.security file to reflect the correct type is handled here. If it fails, upgrade history and agent log will have "Exception thrown attempting FIPS file conversion during upgrade.", and a FipsFileConversionFailure MID Server Issues record created. Upgrade History "MigrateTrustStore" Stage is for preserving the customer-added certificates in the cacerts file. This will usually be Skipped for patch and hotfix upgrades with message "TrustStore migration not necessary (JRE upgrade not required).". Major version upgrades will require this to run. If it works we log "TrustStore migration complete." with "Backup of the original TrustStore can be found at: " just in case it actually didn't, or log "No certificates identified for TrustStore migration." if no certificates had been added to cacerts. A TrustStoreMigrationFailure MID Server Issues record created with any errors.Upgrade History "Deploy Binary Files" Stage is set to In Progress.The "WMI Collector Service" on Windows should no longer exist, but if it does it is stopped now.The warning "Unable to determine upgrade scenario with current=" can be ignored in as it comes from the code specifically for "Detecting down-grade from Rome or later to a version prior to Rome:", which is unlikely to ever happen now.Since Quebec, "Writing MID and wrapper PIDs to ... agent\conf\pids" for checking the Java and Tanuki processes are truly stopped in the later step. It that fails, it logs "Could not write MID processes to file. Proceeding with upgrade without the recorded PIDs."Since Washington DC, the service name is written to a file, and related to preventing privilege escalation. "Writing service name to ...agent\service_name" and if it fails "Could not write service name to file. Proceeding with upgrade without recorded service name.".If a DistUpgrade file is left over from a previous upgrade it is delete. "Attempt to delete file: ...\agent\conf\DistUpgrade" and "File not found or unable to delete" is what you would expect to see, because that's not expected to be there.The JRE (java executable) to be used for the upgrade process is checked, to make sure it is there and usable, and the user has execute permissions. Warnings if this fails would include "Unable to find Java executable for extracted JRE.", "Unable to prepare extracted JRE for upgrade.", and exception, "Cannot determine a current or packaged JRE to upgrade with, caused by: "If the midupgrade.conf is not there, it will be created, or you will see error "Unable to create the upgrade config file to pass parameteres. The process was failed with exception:...". It will then be populated with a list of files that have changed. The contents of the file are "mid.upgrade.replace_changed_files=" followed by a list of filenames. If it is not possible to create that list, warning "Unable to save mid.upgrade.replace_changed_files (...) in <filename> The process was failed with exception:".On Windows you may see "Reinstall Service -- Reinstalling the service is supported only for linux OS", which you can ignore. On Linux, you will see one of these: "ReinstallService -- Reinstalling the service is required. MID Server upgrade continues with re-installing the service", and a Upgrade History "Reinstall Service" Stage is set to In Progress.Warning "ReinstallService -- User is not authorized to reinstall the service during MID upgrade. MID upgrade continues without re-installing the service", and a MID Server issues record containing "MID Server upgrade needs to re-install the service.However, the current user is not authorized to install the service (only root/admin user has privilege). Upgrade continues WITHOUT re-installing the service which is not safe. Reinstalling the service manually is recommended. Make sure that the service is installed with the same version of MID Server wrapper."A MID Server issues record for "Reinstall Service verification was failed during MID Server upgrade. It is possible that a required service re-installation was skipped. Make sure that the Service is installed with the same version as MID Server after upgrade."Or nothing needs doing, and any existing MID Server Issues record is marked as resolved. "Stopping MID server. Bootstrapping upgrade." and "MIDUpgrader.startUpgradeRunner(), OperationalState=UPGRADING" will be written to the MID Server agent log when the upgrade Process is started, which executes the upgrade binaries that are now in the TEMP folder.6.1 Since. The <TEMP>\<random number>-0\<mid buildstamp>\upgrade-wrapper\bin\glide-dist-upgrade.bat file is run to do that. If this fails you may eventually see "Failed to bootstrap."The temp folder name is written to agent\work\upgrade.info , logged as "Added marker `<path>` to upgrade marker file.". If this fails: "Unable to write upgrade marker file `<path>`. Marker `<line>` must be deleted manually."From Tokyo, the MID Server now waits for confirmation that the upgrade process has started, unless MID Sever parameter mid.skip.dist.signal.for.shutdown=true is set. The value of this parameter is logged as "Skip dist signal check: false".This will also log "assignedVersion:" of the version we are upgrading to.A warning "Unable to determine release version with assigned = " is shown if we are not able to work out the Family release from that assignedVersion (using GlideFamilyRelease.releaseIsAfterRelease). This check is done to see if we are pre-Tokyo version, in which case the check is skipped. That doesn't stop the upgrade continuing.The MID Server now checks for the DistUpgrade file, which the upgrade process will write if it has successfully started. While this is not seen yet, every 10 seconds for 10 minutes, it logs "Waiting for upgrade process started signal" and "DistUpgrade file is not present."If after 10 minutes there is still no confirmation, it gives up and logs "Attempt to delete file: ..\agent\conf\DistUpgrade", "File not found or unable to delete" and warning/MID Server issues record "Creating unique MID issue with message "Unable to receive upgrade process status" for source "Couldn't Launch Upgrade Process"" Note: Prior to Orlando, the Process was instead a new Windows Service named "ServiceNow Platform Distribution Upgrade (<MID Server name>)". The MID Server service needed a 'logon as' user that is a member of the local Administrators group, or it will not be able to create and start the temporary upgrade service.Once it has confirmation that the upgrade process has started, "Setting mid status to Down" is logged at the start of the shutdown process. The wrapper log will log "Stopping the ServiceNow MID Server_xxx service..." at this point."MIDServer MID Server stopped" is logged, however that doesn't always mean that all threads have been killed or that the JVM has stopped yet. There will still be probes that have not finished yet, and those are still going to have to end, or may crash with exceptions.6.6 The last agent log entry expected is "Thread-0 Main.handleStop() after shutdown, OperationalState=UPGRADING"During this time wrapper log shows several "Waiting to stop..." logs, and will continue to repeat that every 5 seconds until all running threads/probes have ended.Finally a log of "<-- Wrapper Stopped" in the wrapper log shows the JVM has shut down. There should now be no files locked for the java application, or wrapper service. This will take >2 minutes longer than normal in Madrid6.2 , and if there are other stuck probes this can take 15 minutes or more6.3,7.12. Meanwhile, the separate Upgrade process starts, and will do the following: This will start immediately after the "Bootstrapping upgrade" log above, before the MID Server has finished shutting down, which may take some minutes to complete.This is another Java application, launched using the Tanuki wrapper.This process logs to glide-dist-upgrade.log, within the TEMP folder. Only later is this copied into the main wrapper.log. Any errors or warnings will be logged. If the upgrade fails during this step, then this file may be the only clue as to what happened."The ServiceNow Platform Distribution Upgrade (xxx) service is not installed - The specified service does not exist as an installed service" is logged in Orlando and Paris6.4. Ignore that."com.snc.dist.mid_upgrade.UpgradeMain$1 start" is logged once the java wrapper has started the upgrader application.From Tokyo, a file is written to agent\conf\DistUpgrade, as a way of telling the main MID Server service that it can now shut down.This upgrade process waits until the MID Server service has fully shut down, before continuing7.9. It will first query the Tanuki wrapper every few seconds (using "agent\bin\mid.bat status")until it returns "Running: No", and those results are written directly to the wrapper.log. Since Quebec, it also checks the MID and wrapper process previously recorded in file agent\conf\pids are truly stopped. Expect 2 entries like "Process (pid=xxx) is not running." once both are stopped. Note: If something has started the MID Server service again since it shut down, such as a service monitoring tool, then this will break the upgrade, because the new service will have different PIDs, and we continue even though there will no be file logs from the newly started service.Files in the agent\bin and agent\lib folder are deleted from the MID Server installation. It will retry every second if the file is still locked, so the fact the MID Server might still be shutting down should not be an issue, assuming the MID Server does eventually cleanly shut down. "com.snc.dist.mid_upgrade.UpgradeMain wipeDirs" is logged.If the files are still locked after 10 minutes7.1,7.12 the upgrade will fail. The upgrade service stops, and the MID Server is not started, and remains Down. From Paris, a check that the actual processes have stopped is added 7.14, in addition to logging a list of currently running processes, which will probably confirm the java and wrapper processes were still running. It does not do a stack dump, or list running services, so the information to match up the process IDs (PID) with the installs/services when multiple MID Servers are running is not easy from this log. From Rome, this logging is much improved.If the agent log shows "MID Server stopped" and "Main.handleStop() after shutdown, OperationalState=UPGRADING" it doesn't mean the JVM and wrapper have actually stopped. You need to also see "<-- Wrapper Stopped" in the wrapper.log to confirm the MID Server has shut down. Note: these log entries from the main service are not going to be in chronological order with the upgrade log entries, due to the upgrade log being copied into the wrapper log later.It is possible that file lock errors happen before the 10 minute timeout, after the MID Server truly has shut down. For example, due to Application Experience7.3, and Anti-Virus software such as Cisco AMP7.4 and Dell SecureWorks Red Cloak7.10. There are others causes not yet nailed down7.5. These other non-mid server process are momentarily keeping a lock on the files as the upgrade service tries to delete them. Code is being added to the MID Server to create Issues records when known causes such as these are identified, and Application Experience is already checked for4.6. Anti-virus deleting suspicious files, such InjectorService.exe, while the upgrade is also trying to delete them causes exceptions as well.7.11 The upgrade service stops, and the MID Server is not started, and remains Down.From Rome, only the files that need to be replaced are replaced. This is based on what was recorded earlier in midupgrade.conf. Prior to Rome the whole contents of the bin and lib folders were deleted and replaced even if only a few files had actually changed. For example, the Tanuki Wrapper executable bin\wrapper-windows-x86-64.exe is rarely changed, and so by not touching that many upgrade issues are avoided.If 2 services incorrectly use the same install folder, the copies may fail due to file locks by the other running service.7.2 Checks on MID Server startup should now prevent that possibility.After the deletes are done, "com.snc.dist.mid_upgrade.UpgradeMain migrateToTarget" and "Copying files to MID server installation path" is logged. New files previously extracted into the temp folder are copied into the MID Server installation folder, to replace those just deleted. Any existing files will be overwritten, and so would also need not to be locked.Then "Correcting file permissions for directory" is logged for the correction and enforcement of file and folder permissions, before logging "Finished copying files".If the Java JRE is also upgraded7.6, then the agent\jre folder is deleted and replaced. Customised files within agent/jre such as "cacerts" will be overwritten.7.7,7.13. Once already upgraded to Quebec or later, these should be automatically saved and put into the new JRE's cecerts."Upgrade complete" is loggedA crash or exception around this point could mean no further steps happen. This may be recoverable by allowing the ServiceNow Platform Distribution Upgrade to run again.7.8The main MID Server service is Started using agent\start.bat7.15,7.16"Unable to install the ServiceNow MID Server_xxx service - The specified service already exists. (0x431)" is logged - ignore that, because this is the same start.bat that is used as part of a manual install, and tries to create a service just in case one is not already there. "wrapperm | Waiting to start..." may be logged a few time, and then "ServiceNow MID Server_xxx service started".The log file is copied into the MID Server's wrapper log, in an << UPGRADE LOG BEGIN >>...<< UPGRADE LOG END >> section.This upgrade process then shuts itself down. The MID Server Starts The upgrade check on startup should confirm that the MID Server Installed version is now the Assigned version. If not, it will attempt to upgrade again.The previous ServiceNow Platform Distribution Upgrade Windows service is uninstalled, and the TEMP folder is deleted, even if it had crashed and not finished.8.1The TEMP folder is deleted, and the glide-dist-upgrade.log file with it.The Instance Certificate will be validated, by checking for revocation with OCSP8.2, and the certificate chain and root certificate are also checked, which can cause problems when self-signed certificates of a proxy/firewall are involved.8.10The Tanuki Wrapper will verify the start parameters, and the Certificates of the wrapper executables are valid.8.3,8.4,8.5Passwords in config.xml may be re-encrypted if security-related code has changed.8.6The Powershell version of the host is checked.Cortex XDR has been seen to kill the MID Server Service during startup.8.12A PowerShell script to enforce stricter Windows file permissions is run8.7. MID Server parameter mid.windows_host.file_permissions.enforce=false disables this. In Quebec this can timeout an prevent startup.8.11A check is done to make sure the Service name in wrapper-override.conf matches the actual running service name, and if not shuts down the MID Server to avoid the chance of 2 services for the same install running. 8.8,8.9A check is done for any other MID Server records in the instance with the same name.The Version field of the MID Server record [ecc_agent.version] is updated by the "MID - Process XMLStats" Business rule sensor, in response to the topic=queue.stats input sent by the MID Server's StatusMonitor thread, which runs on startup (and every 10 minutes) and gets the version number from the agent/package/meta/mid-core.meta file.If an external Java (JVM/JDK) is being used (not agent\jre, but a path set in wrapper-override.conf), and is incompatible with the new MID Server version, it may fail to start8.13 Related Links1.1 KB0782134 / PRB1322816 Mid server restart during an instance upgrade coincident with auto upgrade causes MID Server Down2.1 KB0749119 / PRB1344057 MID Server Autoupgrade is not able to handle Instance Rollbacks - MID Server doesn't downgrade again2.2 KB0697389 MID Servers repeatedly Upgrade and Downgrade between current and previous instance version2.3 KB0812239 How does a MID Server know what version it is running?2.4 Docs: MID Server Version Selection3.1 KB0827581 / PRB1409216 While MID server is being upgraded, ThreadPoolExecutor's pause() throws a NPE exception if the worker's source parameter is null or not present and the MID server got stuck in the UPGRADING state4.1 KB0565184 How to upgrade a MID Server that does not have access to the AutoUpgrade install server on the Internet4.2 KB0781701 / PRB1367586 MID Server Pre-Upgrade Check can be blocked for days while running the Powershell test, delaying upgrades4.3 KB0793999 Windows MID Server PowerShell requirements4.4 KB0827961 / PRB1403382 MID Server fails to upgrade from a Signed ZIP file version to a Non-Signed version, because instanceinfo is cached. e.g. MP10 HF1b to NP8, or NP9 to OP24.5 Docs: MID Server pre-upgrade check4.6 KB0858647 / PRB1430215 MID Server can't tell if Application Experience is running on Windows Server 20194.7 PRB1437409 / KB0861832 MID Server pre-upgrade check fails with PSReadLine/PSConsoleReadLine/WinIOError error4.8 KB0995546 Accessing Regulated Market specific Install servers for MID Server download, MID Server Upgrade and Nmap5.1 PRB1332088 Adjustments to the MID Server downloading process - Downloading via instance will no longer be the default for new instances (Since LP8, MP3, N)6.1 KB0747569 How to change the TEMP folder used for MID Server upgrade service and extracted files (Quebec and earlier only)6.2 KB0754285 / PRB1322060 MID Server Stop or Restart takes over 2 minutes longer due to a JMX thread remaining running - "Shutdown failed: Timed out waiting for signal from JVM" 6.3 KB0635920 / PRB1170628 Autoupgrade fails to restart the MID Server successfully in certain conditions (mostly when there are too many running or hung threads) causing the MID to go down6.4 PRB1396562 / KB0861153 MID Server Upgrade log shows error for "The ServiceNow Platform Distribution Upgrade (xxx) service is not installed - The specified service does not exist as an installed service", that does not break anything6.5 PRB1422003 / KB0861154 MID Upgrade does not support changing the temp directory of the JVM (antivirus/security blocking temp folder during upgrade)6.6 KB1002245 / PRB1449497 Null pointer exception when MID Server shuts down after Validating6.7 KB0999273 / PRB1538125 Mid Server fails to check that the temporary Upgrade process is running properly before shutting itself down, causing MID Server Down7.1 KB0749292 / PRB1314105 During the MID upgrade 2 minutes time-out for deleting old files is not enough for some customers7.2 KB0715612 / PRB1307275 MID Server auto-upgrade will fail if Windows has 'Application Experience' disabled7.3 KB0743123 / PRB1330396 MID Server start.bat fails to check if a Windows Service already exists for the installation folder before creating another service7.4 KB0827747 / PRB1408516 MID Server upgrade fails leaving MID Server Down, due to Cisco AMP antivirus preventing the upgrade service deleting the Wrapper executable7.5 KB0715244 / PRB1279578 MID server auto-upgrade fails while copying files from temp to agent folder on Windows even with ''Application Experience' enabled7.6 KB0719830 Is OpenJDK supported as the JRE for MID Servers, and which versions/patches can I use?7.7 KB0750004 / PRB1320637 A MID Server upgrade that includes a new JRE version will overwrite the cacerts file7.8 KB0779816 How to continue a MID Server upgrade after it has crashed in the middle of the ServiceNow Platform Distribution Upgrade service, leaving the MID Server Down and the Service not running7.9 KB0858627 / PRB1419577 The MID Server Upgrade service doesn't properly detect if the MID Server has fully shut down, and so has issues with file locks when it starts deleting, potentially leaving the MID Server down7.10 KB0858554 / PRB1429834 MID Server down after attempted upgrade, due to file locks caused by Dell SecureWorks Red Cloak anti-virus (mid.bat, java.exe, wrapper-windows-x86-64.exe)7.11 PRB1437357 / KB0861733 Upgrade fails leaving MID Server DOWN, due to antivirus/security software flagging InjectorService.exe as Malware/Trojan7.12 PRB1373214 / KB0784943 MID Server Down after auto-Upgrade service 10 minute file lock timeout: SEVERE: java.io.IOException: Unable to delete agent\lib\accessors-smart.jar7.13 PRB1451866 / KB0869024 The fix for PRB1320637 requires that the cacerts Truststore file password remains as the default "changeit", which many customers won't allow, causing certificate deletion during JRE upgrades (e.g. Quebec) and subsequent MID Server and Integration outage7.14 KB0962099 / PRB1498314 MID server upgrade on Japanese Windows server fails leaving it DOWN with: Wrapper.stop() failed to stop the MID Server. Trying to use taskkill command to kill the process7.15 PRB1396562 / KB0861153 Latest Version MID Server Auto-Upgrade fails for JRE 11.0.11+ resulting in server shutdown7.16 KB1001745 / PRB1547917 Before shutting down for an upgrade, MID Server fails to confirm that it will be able to start itself again, or report the issue, leaving the MID Server Down8.1 KB0792621 / PRB1381000 MID Server service starts regardless of whether a failed Upgrade Service still remains, leading to deletion of the upgrade service, and potentially leaving the MID Server running in a semi-upgraded and corrupt state8.2 KB0813636 / PRB1385357 MID Server can fail to install or upgrade to Orlando due to new external connectivity requirement to ocsp.entrust.net for OCSP certification revocation verification check8.3 KB0821436 / PRB1394171 Linux MID server auto-upgrade can fail even on Tanuki wrapper version 3.5.36 if MID service is not reinstalled8.4 KB0746874 / PRB1312206 Linux MID server upgrade fails, and MID Server remains Down, on upgrade to Madrid - Tanuki wrapper service shutdown as 'TERM Trapped'8.5 KB0727033 MID Server won't start due to Root Certificate requirements - wrapper error "The digital signature of the object did not verify"8.6 KB0826461 / PRB1406629 After Upgrade to Orlando, MID Server fails to connect to Instance with "last block incomplete in decryption" if the proxy password is blank8.7 PRB1396279 / KB0858646 File permissions enforcement removes permissions of the windows service's log on as user, when computer or domain name has lowercase letters, leaving MID Server down8.8 PRB1435863 / KB0861098 Mid Server DOWN due to false "expected service name" error, (1) if because of powershell issue the service name cannot be fetched or (2) if the 'case' of the Service Name not exactly matching between wrapper-override.conf and the running service/registry 8.9 PRB1437392 / KB0861829 Mid Server DOWN due to false "expected service name" error, if the checking script has a PSReadLine/PSConsoleReadLine error8.10 PRB1447511 / KB0864241MID Server shows certificate chain errors with self-signed cert8.11 PRB1489024 / KB0959468 In Quebec : MID Server upgrade stops if Powershell tests in Pre-upgradecheck failed and Powerconsole remains busy and the closing the session failed8.12 PRB1496588 / KB0961505 MID server service is terminated unexpectedly on startup (error id 7034), causing MID Server to go Down, due to third party security software Cortex XDR8.13 KB0861153 / PRB1396562 MID Server Auto-Upgrade fails for JRE 11.0.11+ resulting in server shutdown. PRB1484957 Alternate automatic MID upgrade method needed for GCC, SPP (Australia) and future regulated marketsKB1001926 / PRB1548158 32 Bit MID Server installed on 64 Bit host machine fails AutoUpgrade in RomeKB1001748 / PRB1548649 MID Server upgrade carries on with the deletes, when it knows the wrapper is still running, if the previously recorded PIDs are no longer the currently running PIDs, leaving a half-deleted install and the MID Server in Down stateNon-English hosts (i.e. French or Spanish) may cause problems for the regex that processes the result for the PID check for double-checking the mid server is shut down.PRB1516216 MID Server fails to upgrade/downgrade to version that is using old signed certificate - NEEDS KB, >4 cases "Aborting MID Server upgrade due to pre-upgrade check failure: Signature verification failed: Entry doesnt have valid signer"