MID Server Connectivity Issues due to OCSP failures - Known Error

MID Server Connectivity Issues due to OCSP failures Description OCSP failures are one of the leading causes of MID Server connection issues. OCSP is a Hypertext Transfer Protocol (HTTP) used for obtaining the revocation status of an X.509 digital certificate.

After MID starts up, it initiates an SSL handshake with that of the instance. During this handshake, MID performs a real-time status check of service-now certificate with the URI: http://ocsp.entrust.net . It is required to connect with the certificate authority ( CA ) that issued the SSL certificate to ensure that it was not revoked. If MID fails to receive any response or there is a connection failure, then the certificate would be assumed to be revoked, and the MID fails connecting to the instance.

Known Errors OCSP is an industry-standard that is meant to run over Port 80. If firewall rules block port 80 traffic, you will see error in agent logs.

If MID host has port 80 blocked, you may see these errors in MID agent.log Starting from Tokyo release, we display the Module name along with the error messages. For OCSP related errors, GlideSSLProtocolSocketFactory, will throw an IOException as shown below. 1. - WARN (SignatureValidationMonitor.1200) [GlideSSLProtocolSocketFactory:509] OCSP revoke check IOException for *.service-now.com

- [HTTPClient:710] Request not sent to uri= *** = : org.apache.commons.httpclient.HttpException: Permission denied: connect: Permission denied: connect

2. - com.glide.communications.GlideSSLProtocolSocketFactory$OCSPException: Permission denied: connect.

Prior to Tokyo release, we can see following errors in the agent logs. 1. - RefreshMonitor.65 WARNING *** WARNING *** OCSPCheck HTTP status: 503, error message: Method failed: (/) with code: 503

- RefreshMonitor.65 WARNING *** WARNING *** OCSP revoke check IOException for *.service-now.com

2. - (Network Configuration issue) Please check that the MID server can ping the instance: https://xxxx.service-now.com/ You may also need to configure the network that the MID server uses to allow traffic over TCP port 443.

- StartupSequencer SEVERE *** ERROR *** test failure java.lang.IllegalStateException: Unable to connect to instance.

If OCSP responder is not reachable, you may see one of the following errors in MID agent.log - com.glide.communications.GlideSSLProtocolSocketFactory$OCSPException: The host did not accept the connection within timeout of 10000 ms *.service-now.com - com.glide.communications.GlideSSLProtocolSocketFactory$OCSPException: unknown tag 28 encountered

2. - *** WARNING *** OCSPCheck HTTP status: 503, error message: Method failed: (/) with code: 503

- *** WARNING *** OCSP revoke check IOException for *.azurewebsites.net

- *** WARNING *** Request not sent to uri= https://something.azurewebsites.net/api/ :org.apache.commons.httpclient.HttpException: unknown tag 28 encountered

3. - *** WARNING *** OCSPCheck HTTP status: 503, error message: Method failed: (/) with code: 503

- *** WARNING *** OCSP revoke check IOException for *.service-now.com

- *** WARNING *** org.apache.commons.httpclient.HttpException: unknown tag 28 encountered

- StartupSequencer SEVERE *** ERROR *** test failure

- java.lang.IllegalStateException: Unable to connect to instance.

We use third-party HTTP Client library. Sometimes, it throws a meaningful error message like: - “Permission denied: connect”. Other times, we do see HTTP exceptions like “Unknown tag error” or "unknown tag 28 encountered" in the logs. The unknown tag indicates, http client could not figure out the reason for failure. We should ONLY rely on following errors from our code base: -

“OCSP revoke check IOException for *.service-now.com”, indicating failure to connect to service-now.com

Steps to Reproduce

To set up a test to reproduce this on demand against Entrust OCSP responders is probably not possible.

Workaround Workaround Check Connectivity to the OCSP server from MID server host machine. Open a browser on the server hosting the MID Server and try accessing this URL: http://ocsp.entrust.net/

The expected behavior is to see a blank page. If any type of error is present, it is an indication that something is blocking access to the Entrust OCSP site.

At this stage, you should involve your network team, to check if communication on Port 80 is blocked and fix this issue with the help of your network team.

THIS IS NOT A SUGGESTED WORKAROUND: - create a policy for the target url so that the OCSP check is not performed.

NOTE: - Your security team would need to determine what they consider the safest solution out of the two above.

TEMPORARY ONE TIME WORKAROUND: - Prior to Utah release, if the MID Server is down due to OCSP errors, you can set ONE-TIME temporary workaround by setting the following MID Server parameter in its " agent\config.xml”: -

This value will be overwritten by MID security policy values on the instance. By default, service-now.com OCSP value is set to true. So, next time you restart MID, you will still may face the same OCSP issue.

TEMPORARY WORKAROUND FOR ALL MID SERVERS (NOT an recommended approach): - You can navigate to MID Security Policy and turn off OCSP check for either *service-now.com, internet and intranet access. This will sync the policies to all the MID servers, which are currently up and running.

Or you can start with temporary one-time workaround and turn ON/OFF for all mid servers using the MID certificate policies. This policies will automatically sync to all mid servers.

Starting Utah, we are adding a config parameter called "mid.ssl.use.instance.security.policy”. MID server has its choice to use bootstrap parameters over the security policy from instance by utilizing this parameter. Here are its behaviors:

By default, it will be false. When false, it won't sync with the *.servicenow.com policy from instance mid_cert_check_policy table; true, otherwise For every MID server, the user will be able to modify the "mid.ssl.use.instance.security.policy" config parameters. Whenever it changes to true, MID Server will sync all policies from instance and the bootstrap config parameters will be overwritten by the *.servicenow.com policy on instance mid_cert_check_policy table. The final policies will update in policy map in MID memory as well as config.xml file. With that said, here are the default parameters values in config.xml