MID Server can fail to install or upgrade to Orlando due to new external connectivity requirement to ocsp.entrust.net for OCSP certification revocation verification check


Description

The MID Server tries to connect to the ServiceNow instance after the upgrade to Orlando. However, with the high-security OCSP check involved, the host machine is not able to make a successful OCSP check against the ServiceNow certificate to the following URI: http://ocsp.entrust.net

In that case, on upgrade to Orlando or a fresh install of Orlando MID Server, we see one of the following errors in the logs:

"OCSP revoke check IOException for *.service-now.com 
org.apache.commons.httpclient.HttpException: Connection reset"

It may be seen in these threads:

File sync worker: ecc_agent_jar OCSPCheck adding BouncyCastle provider at -1
File sync worker: ecc_agent_jar OCSPCheckedCertificateCache build with max capacity 32
File sync worker: ecc_agent_jar OCSPRevokedCertificateCache build with max capacity 16
File sync worker: ecc_agent_jar WARNING *** WARNING *** Socket error
File sync worker: ecc_agent_jar WARNING *** WARNING *** OCSP revoke check IOException for *.service-now.com
File sync worker: ecc_agent_jar WARNING *** WARNING *** org.apache.commons.httpclient.HttpException: Connection reset
ECCQueueMonitor.1 WARNING *** WARNING *** OCSP revoke check IOException for *.service-now.com 
ECCQueueMonitor.1 WARNING *** WARNING *** org.apache.commons.httpclient.HttpException: OCSP communication error 403 Method failed: (/) with code: 403 - Forbidden username/password combo
StartupSequencer WARNING *** WARNING *** Socket error 
StartupSequencer WARNING *** WARNING *** OCSP revoke check IOException for *.service-now.com
StartupSequencer WARNING *** WARNING *** org.apache.commons.httpclient.HttpException: Connection reset

Steps to Reproduce

On an Orlando Instance:

  1. On a host that does not have access via http/https to OCSP responder "ocsp.entrust.net"

  2. Upgrade a MID Server from Madrid to Orlando, or install a new Orlando MID Server

  3. Start Up the MID Server and see the following error:
    "OCSP revoke check IOException for *.service-now.com org.apache.commons.httpclient.HttpException: Connection reset"

  4. The MID Server will not connect to the instance and be Down

Workaround

This is expected behavior and by design in Orlando. Please review the documentation for more details

There are a number of possible causes for this error, and identifying which will allow you to open the required access:

  1. The Entrust OCSP responder is unavailable.
  2. An internal firewall rule or proxy configuration prevents the OCSP call from going out, and the connection fails.
  3. A web filter and proxy configuration prevents such external sites from being accessed.

Resolution:

  1. In Windows based environments, if the domain controller is behind a firewall, you may have to configure the firewall to explicitly allow outgoing HTTP connections to enable the domain controller to connect to the OCSP responder.
  2. Verify if outgoing traffic to the Entrust OCSP server, from the MID Server host, is blocked by a firewall appliance or a proxy configuration.  In this case, you will have to allow-list the FQDNs of the Entrust responders as required by Entrust. (e.g. crl.entrust.net, crl2.entrust.net, ocsp.entrust.net, although this may change)
    If you are not using the ServiceNow certificate because you are using a vanity URL or your own certificate, then you can confirm the OCSP and CRL URLs using this 3rd party tool, replacing hi.service-now with your instance URL: https://www.ssllabs.com/ssltest/analyze.html?d=hi.service-now.com
  3. Clear the validation endpoints LIST by setting the value of MID Server property of mid.security.validation.endpoints to blank. This will turn off the external endpoint validation. Do not remove the property.
  4. Important: After resolving the issues with the firewall application or proxy configuration, change the value of the 'mid.security.validation.endpoints' property back to '*.servicenow.com'. If you leave either the 'mid.security.validation.endpoints' property removed or value empty, the MID Server will not validate any TLS connections and the MID Server will be vulnerable. If you see a warning message in MID issue table, it means the 'mid.security.validation.endpoints' is not properly configured.

Note: On-prem customers only need to complete step 3 to resolve the issue.

If the resolution fails, try capturing the network traffic using Fiddler/Wireshark when this issue presents and attach it to the case. You can work with your network team for this. OCSP runs over HTTP and it should return a response with certStatus = good, meaning that the certificate is valid for the SSL transaction between the MID Server and the instance.



Related Problem: PRB1385357