Troubleshooting MID Server DownDescriptionThis article is a starting point for troubleshooting MID server down issues: what the MID server status represents, where the MID server logs can be found, and troubleshooting MID server down. MID Server Status The MID server status is not an indication of whether the MID server service is running or stopped. Rather, the MID server status is a representation of the communication between the MID server and the instance. The MID server status is decided based on the last heartbeat input or ecc_queue record inputs received by the instance from a MID server. The scheduled job, or MID Server Monitor checks if a heartbeat input was returned after the last heartbeat output, or if any ecc_queue input was returned by the MID server within the last minute. The MID Server Monitor job calls the MonitorMIDServer script include. The MID server status is set to down if no heartbeat or ecc_queue input was received by the instance from the MID server. The MID server status is set to up the next time the MID Server Monitor job runs if any heartbeat or ecc_queue inputs were received from the MID server. Therefore, a MID server status may be set to down, even if the MID server service is running on the MID server host. The MID server status may be set to down if: The MID server configuration file, agent/config.xml, has incorrect configuration such as invalid credentials, incorrect instance URL, or misconfigured proxy informationThe communication from the MID server to the instance is being blocked, which could be due to a proxy or a firewallThe MID server service is not running, this could be caused by a manual stop or a crash of the MID server service Note: Orlando Patch 7a/8, Paris Patch 1 Hot Fix 3a/5a, Paris Patch 2, and Quebec add code to check certificate chains for all outbound requests using TLS certificates, which may cause errors due to missing certificates in cacerts that had previously gone unchecked. If the MID server connects to the instance through a proxy, the certificates must be imported to the MID server cacerts. For more information on certificates, please see: Troubleshooting MID server SSL issuesMID Servers and CertificatesHow to obtain SSL certificate from the browserAdd SSL certificates for the MID Server MID Server Logs You can find where the MID Server is installed from the Host name and Home directory fields of the MID Server record in the instance. The MID server logs can be found under: <MID_Install_Directory>\agent\logs\ In the logs folder there are several sets of log files, but the most important files are: agent0.log.0wrapper.log Troubleshooting Configuration File Check Configuration File Parameters: The configuration file, config.xml, is found under: <MID_Install_Directory>\agent\ The following is part of an example MID server configuration file: <!-- Tells the MID server where to contact its associated Service-now instance. Edit this value to provide the URL of your organization's Service-now instance. --><parameter name="url" value="https://YOUR_INSTANCE.service-nowcom"/><!-- If your Service-now instance has authentication enabled (the normal case), set these parameters to define the user name and password the MID server will use to log into the instance. --><parameter name="mid.instance.username" value="midserver"/><parameter name="mid.instance.password" value="encrypted:AqSKti1xfos2IuWzkMQTFg==" encrypt="true"/> Check that your configuration file does not have any typos for the parameters. MID Server User Checks Confirm configured user is not locked: On your servicenow instance, navigate to System Security > Users and Groups > UsersSearch for the user id configured as the mid.instance.username, in this case we used midserver however your configuration file will likely have another userCheck that the Locked Out box is not checked Confirm user has the mid_server role: On the user record select the related list RolesConfirm the mid_server role is assigned to the user, assign the role is not Check for authentication errors for the user: Navigate to System Logs > System Log > All and search for Basic authentication failed for user:If any basic authentication errors are found for the MID server user then the password may be incorrect Note: MID Server agent log errors like "Could not authenticate user" would also indicate the password is incorrect If the authentication errors are found perform either A or B as follows: A– Update the password for the user to match what was set on the config file for the MID server. It may be best to reset the user password and then update the configuration file as the MID server encrypts the configuration file password. B– Update the password configured on the MID server service config file: Navigate into the MID server and open <Installation_Directory>/config.xml, open with notepad on windows or vi if in linux, where <Installation_directory> is the location you installed the MID on. Search for mid.instance.password.Replace the <parameter encrypt="true" name="mid.instance.password" value="encrypted:Old_Value"/> with <parameter encrypt="true" name="mid.instance.password" value="newpassword"/>. In the previous step newpassword is the password for the account set for the MID server user. The password will be re-encrypted once the service is restarted. Restart the service. You need to do the above steps for each MID server service. Give a minute and the MID server status should be updated to Up. Network Tests Ping the instance from the MID server: A ping confirms the instance can be reached from the MID server. Log into the MID server host, open up the command prompt and run: ping example.service-now.com Note: Some environments may have ICMP requests disabled which could cause ping to fail. Telnet to the instance: If a ping is successful, then the instance can be reached from the MID server host. However communication on the port used to communicate to the instance may be closed. From the MID server command prompt: telnet <ip_address> 443 If the telnet test fails, an error message would be displayed, consult your network team to confirm: There are no network firewalls blocking traffic from the MID server to the target host on the port tested Log into instance: From the MID server host, open up an internet browser an log into the instance. If login is successful from the browser, however the mid server still cannot communicate to the instance, compare the proxy settings used by the browser to the MID server proxy settings found in the configuration file. Check IE proxy settings: Click Tools, then Internet OptionsSelect the Connections tabClick on LAN Settings The "Local Area Network (LAN) Settings" may be configured to "use automatic configuration script". Such scripts could configure system proxy settings even when "Use a proxy server" checkbox above is unchecked. Review the script to determine if a proxy is being used. Note: If the browser is configured to use a proxy, it is a good indication the MID server needs to be configured to use a proxy as well. When a browser is not available for testing, the command line may be used instead. Windows: Open powershell and run: PS C:\Users\Administrator> (Invoke-WebRequest -URI <url>).StatusDescriptionOK Linux: From terminal run: [user@Server]# curl -s -o /dev/null -w '%{http_code}' <instance_address>; echo ""; 200 InstanceInfo The first page requested by the MID server from the instance to test connectivity is the InstanceInfo page. You can simulate this request. Linux curl https://<InstanceName>.service-now.com/InstanceInfo.do?SOAP --user <userName>:<userPassword> --header 'Content-Type: text/xml;charset=UTF-8' --header 'SOAPAction: Get' --header 'user-agent: internal_soap_client' --data '<SOAP-ENV:Envelope xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:SOAP-ENC="http://schemas.xmlsoap.org/soap/encoding/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:tns="http://www.service-now.com/GetMIDInfo" xmlns:m="http://www.service-now.com" xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/" SOAP-ENV:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/"><SOAP-ENV:Body><m:execute><table xsi:type="xsd:string">ecc_agent_log</table></m:execute></SOAP-ENV:Body></SOAP-ENV:Envelope>' | xmllint --format - Windows Via an application like PostmanSet basic authentication and provide user name and passwordUse URL: https://<instance>.service-now.com/InstanceInfo.do?SOAP HTTP Headers: Content-Type: text/xml;charset=UTF-8 user-agent: internal_soap_client SOAPAction: Get Body: <SOAP-ENV:Envelope xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:SOAP-ENC="http://schemas.xmlsoap.org/soap/encoding/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:tns="http://www.service-now.com/GetMIDInfo" xmlns:m="http://www.service-now.com" xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/" SOAP-ENV:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/"><SOAP-ENV:Body><m:execute><table xsi:type="xsd:string">ecc_agent_log</table></m:execute></SOAP-ENV:Body></SOAP-ENV:Envelope> Check Service Status Windows: The service status of a MID server installed on a windows host can be checked under windows services as follows: Open windows server managerClick Tools and select ServicesFind the ServiceNow MID Server serviceConfirm its status is Running Linux: The service can be confirmed to be running in linux by one of the following methods: From the agent folder run the start.sh, ServiceNow MID Server is already running will be displayed if the service is already runningtail the MID server log to confirm output is being writtenps -ef | grep "agent/" Review Logs: If all tests are successful, and still the MID server is down, review the MID server logs for more information on why the MID server is down. Release or EnvironmentAll currently supported releases.