Troubleshoot MID Server upgrade issuesDescription Table of Contents Troubleshooting | SymptomsTroubleshooting | DetailsTroubleshooting | Questions and Answers Was the instance recently upgraded either manually or with a QPP patch?Is the MID Server up?Can you read the agent log?"The MID Server was unable to download""Unable to delete file""User is unable to authenticate"File not foundNo, the instance was not recently upgraded Troubleshooting | Symptoms MID Server status is Down in the MID Server listDiscovery scans get stuckMID Server does not keep runningMID Server status is Up but is not responding Troubleshooting | Details When an instance is upgraded, any MID Server pointing to that instance will attempt to auto-upgrade to the same version. The Quarterly Patching Program (QPP), which started in early 2016, forces instances to upgrade to the next patch. The patches are in the same release family (for example, Geneva Patch 1 to Geneva Patch 12) and will not upgrade outside of the release family (for example, Geneva Patch 1 will not auto-upgrade to Helsinki Patch 4). This means that MID Servers attached to instances will be auto-upgrading at least once per quarter. The increased frequency of upgrades might lead to uncommon and unexpected conditions on the machines where the MID Server is installed. Troubleshooting | Questions and Answers Was the instance recently upgraded either manually or with a QPP patch? I don't know To find out if your instance has been updated recently: Navigate to System Diagnostics > Upgrade History.Search the To column for an entry that indicates that the instance was upgraded to a new patch. It will look something like this: glide-RELEASE_FAMILY-03-09-2015_06-22-2016_1932.zip Yes, the instance was recently upgraded See the MID Server logs to determine the state of the upgrade. Is the MID Server up? I don't know Navigate to MID server > Servers.Locate the MID Server in the list. It will report Up with a green dot if it is running. Navigate to MID server > Servers.Locate the MID Server in the list.Open the Mid Server record.In the Related Links form section, select Grab Mid Logs.The ECC Queue list will display two log commands: one for agent0.log.0 and one for wrapper.log.When the requests return, open up the ECC Queue record and download the agent log. No, the MID Server is not up In order to assess the state of the MID Server upgrade with the down MID Server: Log on to the host machine the Mid Server is running on.In the file system, navigate to the Agent > Logs directory.Open the agent log. Yes, the MID server is up Can you read the agent log? Yes, I can read the agent log. Open the agent log file and search for the following phrases: "The MID Server was unable to download" ROOT CAUSE: MID Server unable to download is an indication that the communication between the MID and the instance has been disrupted.SOLUTION: Troubleshoot the communication between the MID and the Instance and try again to upgrade the MID server. "Unable to delete file" ROOT CAUSE: As part of the upgrade process, the MID Server needs to remove and replace certain files. If for some reason one of those files cannot be deleted, the MID Server upgrade fails. Unfortunately, at the time of the Istanbul release, there is no way to recover and proceed with the upgrade. The MID server must be re-installed once the issue with the file system has been resolved.SOLUTION: Resolve local environment issues of the MID Server host and try again to upgrade the MID Server. "User is unable to authenticate" ROOT CAUSE: All MID Servers are authenticated by the user and password that is entered into the config.xml file on the MID Server host machine. The authentication information is stored in the parameter's mid.instance.username and mid.instance.password. The user in this configuration file must also exist on the instance in the System > Users table. The MID user must have, as a minimum, the mid_server role. Failure to authenticate during an upgrade can hang both the MID and the Upgrade Service itself.SOLUTION: Troubleshoot MID Server credentials and try again to upgrade the MID Server. File not found "SEVERE:com.snc.dist.mid_upgrade.UpgradeExecption: java.io.FileNotFoundException:" AND"\bin\mid.bat (Access is denied)" ROOT CAUSE: A complete example containing the two searchable phrases provided is: com.snc.dist.mid_upgrade.UpgradeException: java.io.FileNotFoundException: D:\ServiceNow\agent\bin\mid.bat (Access is denied). This message can indicate that the host machine the MID Server is on is a Windows Machine with Application Experience disabled. Per Microsoft's security documentation, this service is internal and cannot actually ever be disabled. Setting this to disabled in the Windows environment only stops the service from being called. It does not stop the service from running. The purpose of the Application Experience, generally speaking, is to evaluate incoming compatibility of application updates to existing installed software. Since the running service does not receive the request, the evaluation never happens and the install of the update fails. This is what is happening to the MID Server.SOLUTION: Resolve local environment issues of the MID Server host and try again to upgrade the MID Server. No, I cannot read the agent log. If you have not been able to gain access to the agent log following this troubleshooting guide, contact your network or system administrator. The MID Server host machine might be powered off or inadvertently off-network. Resolve the network or hardware issues and try again to upgrade the MID Server. No, the instance was not recently upgraded Continue debugging MID ServerMID Servers and Certificates