How to upgrade a MID Server that does not have access to the AutoUpgrade install server on the InternetDescriptionIf you need to upgrade a MID Server that does not have access to the AutoUpgrade install server, this article provides a manual procedure. The steps below fool the MID Server into thinking it has downloaded the files already, allowing it to upgrade itself in the normal way. If necessary, this must be done for every MID Server after every upgrade or patch of every instance, without fail. When an instance is upgraded, the MID Server needs to upgrade itself to match and needs to Download Upgrade Files from https://install.service-now.com/ and if it does not have access, it fails to upgrade. The automatic Test MID Server connectivity feature will check for and notify you if the MID Server can't. Warning: A MID Server on the wrong version can cause code and data mismatches between MID Server and instance, potentially causing the MID Server to fail to process commands sent it by the instance, or the instance to not process data coming back from the MID Server. APIs related to Validation and encryption Keychains may also not match. Symptom example MID Server Agent log: AutoUpgrade.3600 Performing pre-upgrade validation tests.AutoUpgrade.3600 Downloading from https://install.service-now.com/glide/distribution/builds/package/mid-upgrade/2019/07/16/mid-upgrade.mid.newyork-06-26-2019__patch0-hotfix2-07-10-2019_07-16-2019_1636.preUpgradeCheck.zipAutoUpgrade.3600 WARNING *** WARNING *** java.net.SocketTimeoutException: connect timed out when posting to https://install.service-now.com/glide/distribution/builds/package/mid-upgrade/2019/07/16/mid-upgrade.mid.newyork-06-26-2019__patch0-hotfix2-07-10-2019_07-16-2019_1636.preUpgradeCheck.zipAutoUpgrade.3600 SEVERE *** ERROR *** java.net.SocketTimeoutException: connect timed out when posting to https://install.service-now.com/glide/distribution/builds/package/mid-upgrade/2019/07/16/mid-upgrade.mid.newyork-06-26-2019__patch0-hotfix2-07-10-2019_07-16-2019_1636.preUpgradeCheck.zipAutoUpgrade.3600 Downloading from http://install.service-now.com/glide/distribution/builds/package/mid-upgrade/2019/07/16/mid-upgrade.mid.newyork-06-26-2019__patch0-hotfix2-07-10-2019_07-16-2019_1636.preUpgradeCheck.zipAutoUpgrade.3600 WARNING *** WARNING *** org.apache.commons.httpclient.ConnectTimeoutException: The host did not accept the connection within timeout of 10000 ms when posting to http://install.service-now.com/glide/distribution/builds/package/mid-upgrade/2019/07/16/mid-upgrade.mid.newyork-06-26-2019__patch0-hotfix2-07-10-2019_07-16-2019_1636.preUpgradeCheck.zipAutoUpgrade.3600 SEVERE *** ERROR *** org.apache.commons.httpclient.ConnectTimeoutException: The host did not accept the connection within timeout of 10000 ms when posting to http://install.service-now.com/glide/distribution/builds/package/mid-upgrade/2019/07/16/mid-upgrade.mid.newyork-06-26-2019__patch0-hotfix2-07-10-2019_07-16-2019_1636.preUpgradeCheck.zipAutoUpgrade.3600 SEVERE *** ERROR *** Aborting MID Server upgrade due to pre-upgrade check failure: Unable to download updates from install serverAutoUpgrade.3600 Setting mid status to Upgrade FailedAutoUpgrade.3600 Instance.updateAgentRecordState(), OperationalState=UPGRADE_FAILED The MID Server form and Issues table will also repeat the error: Aborting MID Server upgrade due to pre-upgrade check failure: Unable to download updates from install server Release or EnvironmentAll versions with Windows or Linux MID Servers.CauseIn the following situations, your MID Server computer has no access to the install server and cannot auto-upgrade itself. You should try to resolve these configuration issues internally first, as these are our documented connectivity requirements: The instance is on-premise and installed inside the customer network (with no Internet access), and the MID Server also has no internet access at all.The instance is hosted in our datacenter, but although the MID Server does have access to the instance, you have not yet arranged for it to have access to our upgrade server: https://install.service-now.com/ResolutionThis manual procedure fools the MID Server into thinking it has downloaded the files itself already, allowing it to upgrade itself in the normal way, and if necessary must be done for every MID Server after every upgrade or patch of every instance, without fail. 1) Find the filenames from the Agent log On the MID Server computer, check the latest 'AutoUpgrade' or 'StartupSequencer' thread entries in the Agent Log for the "Missing:" ZIP file names:<install folder>\agent\logs\agent0.log.0 AutoUpgrade.3600 Current packages: AutoUpgrade.3600 Installed: [mid-core.kingston-10-17-2017__patch0-11-06-2017_11-11-2017_1422.universal.universal.zip, mid-jre.kingston-10-17-2017__patch0-11-06-2017_11-11-2017_1422.windows.x86-64.zip] AutoUpgrade.3600 Assigned: [mid-upgrade.newyork-06-26-2019__patch0-hotfix2-07-10-2019_07-16-2019_1636.universal.universal.zip, mid-core.newyork-06-26-2019__patch0-hotfix2-07-10-2019_07-16-2019_1636.universal.universal.zip, mid-jre.newyork-06-26-2019__patch0-hotfix2-07-10-2019_07-16-2019_1636 .windows.x86-64.zip] AutoUpgrade.3600 Missing: [mid-upgrade.newyork-06-26-2019__patch0-hotfix2-07-10-2019_07-16-2019_1636.universal.universal.zip, mid-core.newyork-06-26-2019__patch0-hotfix2-07-10-2019_07-16-2019_1636.universal.universal.zip, mid-jre.newyork-06-26-2019__patch0-hotfix2-07-10-2019_07-16-2019_1636 .windows.x86-64.zip] AutoUpgrade.3600 Downloaded: [] In that example, the MID Server is still Kingston Patch 0, even though the instance is already upgraded to New York Patch 0 Hotfix 2, and the three missing files in this particular example are: mid-upgrade.newyork-06-26-2019__patch0-hotfix2-07-10-2019_07-16-2019_1636.universal.universal.zipmid-jre.newyork-06-26-2019__patch0-hotfix2-07-10-2019_07-16-2019_1636.windows.x86-64.zipmid-core.newyork-06-26-2019__patch0-hotfix2-07-10-2019_07-16-2019_1636.universal.universal.zip Note: The file names and URLs you need will be specific to a particular version of the platform, that is running in your specific instance. 2) Figure out the full URL for those files The attached Excel Spreadsheet provides the links to the files for a specific version. Use this link to avoid the Viewer loading and turning it into a PDF: /sys_attachment.do?sys_id=9229d73a4780f1d011eaf24c736d43cb You can find the current MID Buildstamp of your instance, which is the version the MID Server should upgrade to, on the Stats page:https://<instance_name>.service-now.com/stats.do Paste it into the attached spreadsheet, and find the URLs for the files identified above. The filenames are very similar, so be careful to select the correct file (e.g. mid-upgrade) and architecture (e.g. windows.x64): Note: If you are running a version earlier than Paris, Orlando Patch 3, New York Patch 9, and Madrid Patch 10 Hot Fix 1b, then you will need to use the Unsigned ZIP Files lower down the spreadsheet. Otherwise use the Signed ones. 3) Download these files manually on another computer that has internet access and then copy those ZIP files to the MID Server folder <install folder>\agent\package\incoming Note: You may be tempted to make things simpler by adding both the Windows and Linux files, so you copy the same set of files to all MID Servers regardless of the OS. Don't do that, because the MID Server may still extract them all, overwriting the correct files with the wrong ones, and breaking the upgrade. 4) Prevent the Pre-Check being run If you see the following error in the agent log, disable the MID Server "preUpgradeCheck" by adding the MID Server parameter mid.upgrade.run_precheck=false to each MID Server that does not have access to the install server. AutoUpgrade.3600 SEVERE *** ERROR *** Aborting MID Server upgrade due to pre-upgrade check failure: Unable to download updates from install server See Disabling the pre-upgrade check in the docs. 5) Restart the MID Server At this point, you can wait for the AutoUpgrade thread to run again (it is on a 1-hour interval) or restart the MID Server service to force it to upgrade now.The next time the AutoUpgrade thread runs, Downloaded: shows the files present in the <install folder>\agent\logs\agent0.log.0 log. It then goes ahead and does the upgrade. StartupSequencer Downloaded: [mid-upgrade.newyork-06-26-2019__patch0-hotfix2-07-10-2019_07-16-2019_1636.universal.universal.zip, mid-core.newyork-06-26-2019__patch0-hotfix2-07-10-2019_07-16-2019_1636.universal.universal.zip, mid-jre.newyork-06-26-2019__patch0-hotfix2-07-10-2019_07-16-2019_1636 .windows.x86-64.zip] You can then re-use the downloaded files for any other MID Servers that are also connecting to the same instance.Additional InformationFor post-Madrid releases, this should not be an issue due to defaulting to using the instance as a kind of proxy (mid.download.through.instance=true), meaning direct access to the install server was not necessary. Since New York, the MID Server once again has to have access to the install server (PRB1332088/PRB627019). It is recommended that the property is set to false, and access is provided to the install server, in order to avoid API_INT semaphore exhaustion, MID Servers failing to connect to the instance and shutting down, and extended upgrade times during instance upgrades.