Troubleshooting MID Server connectivity with the instanceSummaryThis article describes the steps to troubleshoot a down mid server in the following scenario: MID Server service is up and running;Customer can connect to the instance from the MID Server host via a browserCustomer can ping to the instance from the MID Server hostNo Windows firewall is configuredProxy may be configured on the MID ServerInstructions1. Check the MID Server agent log and you should see the following errors: 10/01/19 09:07:23 (175) StartupSequencer WARNING * WARNING org.apache.commons.httpclient.ConnectTimeoutException: The host did not accept the connection within timeout of 10000 ms when posting to https://<instance>.service-now.com/InstanceInfo.do?SOAP10/01/19 09:07:23 (175) StartupSequencer SEVERE ERROR SOAP Request: <SOAP-ENV:Envelope xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:SOAP-ENC="http://schemas.xmlsoap.org/soap/encoding/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:tns="http://www.service-now.com/GetMIDInfo" xmlns:m="http://www.service-now.com" xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/" SOAP-ENV:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/"><SOAP-ENV:Body><m:execute></m:execute></SOAP-ENV:Body></SOAP-ENV:Envelope>10/01/19 09:07:23 (175) StartupSequencer SEVERE ERROR SOAP Response: Status code=0, Response body=null10/01/19 09:07:23 (175) StartupSequencer SEVERE ERROR Problem invoking InstanceInfo on https://<instance>.service-now.com/: Please check that the InstanceInfo page exists in the sys_public table and active="true".10/01/19 09:07:23 (175) StartupSequencer SEVERE ERROR org.apache.commons.httpclient.ConnectTimeoutException: The host did not accept the connection within timeout of 10000 ms when posting to https://<instance>.service-now.com/InstanceInfo.do?SOAP(Network Configuration issue) Please check that the MID server can ping the instance: https://<instance>.service-now.com/You may also need to configure the network that the MID server uses to allow traffic over TCP port 443.10/01/19 09:07:23 (175) StartupSequencer SEVERE ERROR * test failurejava.lang.IllegalStateException: Unable to connect to instance.at com.service_now.mid.services.StartupSequencer.runTests(StartupSequencer.java:386)at com.service_now.mid.services.StartupSequencer$Starter.run(StartupSequencer.java:349) 2. Check SOAP calls: During start up sequence, the MID Server checks connectivity with the instance by invoking a SOAP call to the url below where <instance> is the target instance name: https://<instance>.service-now.com/InstanceInfo.do?SOAP By browsing the above link you should receive the following response: <SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/"> <SOAP-ENV:Header/> <SOAP-ENV:Body> <SOAP-ENV:Fault> <faultcode>SOAP-ENV:Server</faultcode> <faultstring> Error completing SOAP request - Only HTTP POST supported </faultstring> <detail>Error completing SOAP request</detail> </SOAP-ENV:Fault> </SOAP-ENV:Body> </SOAP-ENV:Envelope> This is the expected response since InstanceInfo.do?SOAP doesn't accept GET requests. To view the above message in the browser, make sure that friendly message display is turned off. This simple test however is not very comprehensive as we are still using the browser to contact the instance. Some firewalls or proxy servers may allow traffic initiated from a browser and block everything else by filtering by "user-agent" or any other relevant field in the HTTP header. In this case, testing the instance's url or even testing the SOAP page above will work fine but still the MID Server won't connect. You can use a SOAP client tool such as Postman to perform a SOAP post call to the instance by supplying the following information: HTTP URL: https://<instance>.service-now.com/InstanceInfo.do?SOAP HTTP Headers: Content-Type: text/xml;charset=UTF-8 user-agent: internal_soap_client SOAPAction: Get HTTP Body: <SOAP-ENV:Envelope xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:SOAP-ENC="http://schemas.xmlsoap.org/soap/encoding/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:tns="http://www.service-now.com/GetMIDInfo" xmlns:m="http://www.service-now.com" xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/" SOAP-ENV:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/"><SOAP-ENV:Body><m:execute><table xsi:type="xsd:string">ecc_agent_log</table></m:execute></SOAP-ENV:Body></SOAP-ENV:Envelope> You should receive the following response with tags containing the information from your instance: <?xml version="1.0" encoding="UTF-8"?> <SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> <SOAP-ENV:Body> <executeResponse xmlns="http://www.service-now.com"> <result> <install_name>XXXX</install_name> <instance_name>myinstance</instance_name> <instance_id>XXXX</instance_id> <build_date>XXXX</build_date> <build_tag>XXXX</build_tag> <system_id>XXXX</system_id> <node_id>XXXX</node_id> <instance_ip>XXXX</instance_ip> <mid_buildstamp>mXXXX</mid_buildstamp> <mid_version>XXXX</mid_version> </result> </executeResponse> </SOAP-ENV:Body> </SOAP-ENV:Envelope> If your MID Server is running on Linux you can also curl as follows and the output (formatted with xmllint) should be the same as above: curl --header 'Content-Type: text/xml;charset=UTF-8' --header 'SOAPAction: Get' --header 'user-agent: internal_soap_client' --data '<SOAP-ENV:Envelope xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:SOAP-ENC="http://schemas.xmlsoap.org/soap/encoding/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:tns="http://www.service-now.com/GetMIDInfo" xmlns:m="http://www.service-now.com" xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/" SOAP-ENV:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/"><SOAP-ENV:Body><m:execute><table xsi:type="xsd:string">ecc_agent_log</table></m:execute></SOAP-ENV:Body></SOAP-ENV:Envelope>' https://myinstance.service-now.com/InstanceInfo.do?SOAP | xmllint --format - 3. Check MID Server proxy settings in the file /agent/config.xml <parameter name="mid.proxy.use_proxy" value="true"/><parameter name="mid.proxy.host" value="proxy.company.net"/><parameter name="mid.proxy.port" value="803"/> You should be able to input the proxy information parameters into a browser and test connectivity in step 2: https://<instance>.service-now.com/InstanceInfo.do?SOAP 4. Check the Scripted Web Services "InstanceInfo" for any customization: https://<instance>.service-now.com/nav_to.do?uri=sys_web_service.do?sys_id=0cbfe98d83301000dada83ec37d929c8 5. SSL troubleshooting: You can find the IP address of the instance by following the steps from KB0538621 or you can retrieve the IP with the commands below from your Mac: ping instance_name dig +noall +answer instance_name If you suspect of an SSL negotiation issue, you can enable SSL debugging as follows: In the MID Server, open up the file ../agent/conf/wrapper-override.conf and make sure you have the following line at the bottom of the file (Important! remove this line and restart the MID Server when you complete investigation): wrapper.java.additional.3=-Djavax.net.debug=ssl Restart the MID Server serviceThe ssl debugging information will be logged in the wrapper.log file, the following information is relevant to look for: trustStore is: <Install>\agent\jre\lib\security\cacerts TLS protocol version (TLSv1.2) Cipher Suites in ClientHello Cipher Suites in ServerHello If you run a Wireshark packet capture on the mid server host you can see the ssl session handshake. You can filter by ip address of the instance. At a high level, the ssl handshake should be as follows: MID Server Client Hello ->Instance<- Server Hello<- Certificate<- Server Key Exchange<- Hello DoneClient Key Exchange ->Change Cypher Spec ->Encrypted Handshake Message -><- Change Cypher Spec<- Encrypted Handshake Message<- Application Data ->