Troubleshooting MID Server connectivity with the instance


Description

This article describes the steps to troubleshoot a down mid server in the following scenario:

Instructions

1. Check the MID Server agent log and you should see the following errors:

10/01/19 09:07:23 (175) StartupSequencer WARNING * WARNING org.apache.commons.httpclient.ConnectTimeoutException: The host did not accept the connection within timeout of 10000 ms when posting to https://<instance>.service-now.com/InstanceInfo.do?SOAP
10/01/19 09:07:23 (175) StartupSequencer SEVERE
ERROR SOAP Request: <SOAP-ENV:Envelope xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:SOAP-ENC="http://schemas.xmlsoap.org/soap/encoding/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:tns="http://www.service-now.com/GetMIDInfo" xmlns:m="http://www.service-now.com" xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/" SOAP-ENV:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/"><SOAP-ENV:Body><m:execute></m:execute></SOAP-ENV:Body></SOAP-ENV:Envelope>
10/01/19 09:07:23 (175) StartupSequencer SEVERE
ERROR SOAP Response: Status code=0, Response body=null
10/01/19 09:07:23 (175) StartupSequencer SEVERE
ERROR Problem invoking InstanceInfo on https://<instance>.service-now.com/: Please check that the InstanceInfo page exists in the sys_public table and active="true".
10/01/19 09:07:23 (175) StartupSequencer SEVERE
ERROR org.apache.commons.httpclient.ConnectTimeoutException: The host did not accept the connection within timeout of 10000 ms when posting to https://<instance>.service-now.com/InstanceInfo.do?SOAP
(Network Configuration issue) Please check that the MID server can ping the instance: https://<instance>.service-now.com/
You may also need to configure the network that the MID server uses to allow traffic over TCP port 443.
10/01/19 09:07:23 (175) StartupSequencer SEVERE
ERROR * test failure
java.lang.IllegalStateException: Unable to connect to instance.
at com.service_now.mid.services.StartupSequencer.runTests(StartupSequencer.java:386)
at com.service_now.mid.services.StartupSequencer$Starter.run(StartupSequencer.java:349)

2. Check SOAP calls:

During start up sequence, the MID Server checks connectivity with the instance by invoking a SOAP call to the url below where <instance> is the target instance name:

https://<instance>.service-now.com/InstanceInfo.do?SOAP

By browsing the above link you should receive the following response:


<SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/">
<SOAP-ENV:Header/>
<SOAP-ENV:Body>
<SOAP-ENV:Fault>
<faultcode>SOAP-ENV:Server</faultcode>
<faultstring>
Error completing SOAP request - Only HTTP POST supported
</faultstring>
<detail>Error completing SOAP request</detail>
</SOAP-ENV:Fault>
</SOAP-ENV:Body>
</SOAP-ENV:Envelope>
 
This is the expected response since InstanceInfo.do?SOAP doesn't accept GET requests. To view the above message in the browser, make sure that friendly message display is turned off.
 
This simple test however is not very comprehensive as we are still using the browser to contact the instance. Some firewalls or proxy servers may allow traffic initiated from a browser and block everything else by filtering by "user-agent" or any other relevant field in the HTTP header. In this case, testing the instance's url or even testing the SOAP page above will work fine but still the MID Server won't connect. You can use a SOAP client tool such as Postman to perform a SOAP post call to the instance by supplying the following information:
 
HTTP URL:
 
https://<instance>.service-now.com/InstanceInfo.do?SOAP
 
HTTP Headers:
 
Content-Type: text/xml;charset=UTF-8
user-agent: internal_soap_client
SOAPAction: Get
 
HTTP Body:
 
<SOAP-ENV:Envelope xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:SOAP-ENC="http://schemas.xmlsoap.org/soap/encoding/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:tns="http://www.service-now.com/GetMIDInfo" xmlns:m="http://www.service-now.com" xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/" SOAP-ENV:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/"><SOAP-ENV:Body><m:execute><table xsi:type="xsd:string">ecc_agent_log</table></m:execute></SOAP-ENV:Body></SOAP-ENV:Envelope>
 
 
 
You should receive the following response with tags containing the information from your instance:
 
<?xml version="1.0" encoding="UTF-8"?>
<SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <SOAP-ENV:Body>
    <executeResponse xmlns="http://www.service-now.com">
      <result>
        <install_name>XXXX</install_name>
        <instance_name>myinstance</instance_name>
        <instance_id>XXXX</instance_id>
        <build_date>XXXX</build_date>
        <build_tag>XXXX</build_tag>
        <system_id>XXXX</system_id>
        <node_id>XXXX</node_id>
        <instance_ip>XXXX</instance_ip>
        <mid_buildstamp>mXXXX</mid_buildstamp>
        <mid_version>XXXX</mid_version>
      </result>
    </executeResponse>
  </SOAP-ENV:Body>
</SOAP-ENV:Envelope>
 
If your MID Server is running on Linux you can also curl as follows and the output (formatted with xmllint) should be the same as above:
 
curl --header 'Content-Type: text/xml;charset=UTF-8' --header 'SOAPAction: Get' --header 'user-agent: internal_soap_client' --data '<SOAP-ENV:Envelope xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:SOAP-ENC="http://schemas.xmlsoap.org/soap/encoding/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:tns="http://www.service-now.com/GetMIDInfo" xmlns:m="http://www.service-now.com" xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/" SOAP-ENV:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/"><SOAP-ENV:Body><m:execute><table xsi:type="xsd:string">ecc_agent_log</table></m:execute></SOAP-ENV:Body></SOAP-ENV:Envelope>' https://myinstance.service-now.com/InstanceInfo.do?SOAP | xmllint --format -

3. Check MID Server proxy settings in the file /agent/config.xml

<parameter name="mid.proxy.use_proxy" value="true"/>
<parameter name="mid.proxy.host" value="proxy.company.net"/>
<parameter name="mid.proxy.port" value="803"/>

You should be able to input the proxy information parameters into a browser and test connectivity in step 2:

https://<instance>.service-now.com/InstanceInfo.do?SOAP

4. Check the Scripted Web Services "InstanceInfo" for any customization:

https://<instance>.service-now.com/nav_to.do?uri=sys_web_service.do?sys_id=0cbfe98d83301000dada83ec37d929c8

5. SSL troubleshooting:

You can find the IP address of the instance by following the steps from KB0538621 or you can retrieve the IP with the commands below from your Mac:

ping instance_name

dig +noall +answer instance_name

If you suspect of an SSL negotiation issue, you can enable SSL debugging as follows:

wrapper.java.additional.3=-Djavax.net.debug=ssl

trustStore is: <Install>\agent\jre\lib\security\cacerts

TLS protocol version (TLSv1.2)

Cipher Suites in ClientHello

Cipher Suites in ServerHello

If you run a Wireshark packet capture on the mid server host you can see the ssl session handshake. You can filter by ip address of the instance. At a high level, the ssl handshake should be as follows:

MID Server Client Hello ->Instance
<- Server Hello
<- Certificate
<- Server Key Exchange
<- Hello Done
Client Key Exchange ->
Change Cypher Spec ->
Encrypted Handshake Message ->
<- Change Cypher Spec
<- Encrypted Handshake Message
<- Application Data ->