MID Server Down - OCSP Revocation check fails to connect to ocsp.digicert.com - Support and Troubleshooting

MID Server Down - OCSP Revocation check fails to connect to ocsp.digicert.com Issue MID Servers may go Down, and stop connecting to the instance, with this OCSP Check error seen in the agent log.

This example is the key lines of a Washington DC Patch 7 MID Server's agent0.log.0 file, after startup.

This will retry for 10 minutes:

... INFO (MIDServer) [Main:275] Initializing MID Server ... DEBUG (MIDServer) [MIDTLSPolicy:77] MIDSecPolicy: Certificate revocation check for host[ .service-now.com] is true INFO (MIDServer) [OCSPCheck:166] OCSPCheck adding BouncyCastle provider at -1 INFO (MIDServer) [OCSPCheck:174] OCSPCheckedCertificateCache build with max capacity 32 INFO (MIDServer) [OCSPCheck:188] OCSPRevokedCertificateCache build with max capacity 16 INFO (MIDServer) [OCSPCheck:202] OCSPTimeoutErrorCache build with max capacity 16 WARN (glide.lock.cleaner) [LockManager:86] Freed stuck JVM lock: OCSPCheck CN: *.service-now.com WARN (MIDServer) [OCSPCheck:544] OCSPCheck connection timeout: http://ocsp.digicert.com WARN (MIDServer) [HTTPClient:830] org.apache.commons.httpclient.HttpException: The host did not accept the connection within timeout of 10000 ms *.service-now.com ERROR (MIDServer) [RemoteGlideRecord:918] getRecords failed (org.apache.commons.httpclient.HttpException: The host did not accept the connection within timeout of 10000 ms *.service-now.com) WARN (MIDServer) [RetryExecutor:114] MIDRemoteGlideRecord.query failed with error: org.apache.commons.httpclient.HttpException: The host did not accept the connection within timeout of 10000 ms *.service-now.com, retrying in 10 seconds ... ERROR (MIDServer) [MIDRemoteGlideRecord:174] org.apache.commons.httpclient.HttpException: The host did not accept the connection within timeout of 10000 ms *.service-now.com com.service_now.mid.MIDRemoteGlideRecord$RetryableException: org.apache.commons.httpclient.HttpException: The host did not accept the connection within timeout of 10000 ms *.service-now.com Then the following will then also loop indefinitely, giving the misleading error that the instance user has a problem . That's just a side-effect of it not being able to run the user check because it cannot connect to the instance.

INFO (MIDServer) [Main:275] Initializing MID Server ... WARN (StartupSequencer) [HTTPClient:828] Request not sent to uri= https:// .service-now.com/MIDServerCheck.do?SOAP : org.apache.commons.httpclient.HttpException: The host did not accept the connection within timeout of 10000 ms *.service-now.com: The host did not accept the connection within timeout of 10000 ms *.service-now.com ERROR (StartupSequencer) [InstanceSOAPClient:139] SOAP Request: mid_server,soap,soap_script,soap_query,soap_create,soap_delete,soap_ecc,soap_script,soap_update ERROR (StartupSequencer) [InstanceSOAPClient:139] SOAP Response: Status code=0, Response body=null WARN (StartupSequencer) [UserConfigTest:44] Could not check roles on instance : Request not sent to uri= https:// .service-now.com/MIDServerCheck.do?SOAP : org.apache.commons.httpclient.HttpException: The host did not accept the connection within timeout of 10000 ms *.service-now.com ERROR (StartupSequencer) [StartupSequencer:726] test failure java.lang.IllegalStateException: User: cannot be authenticated. User does not exist, or missing the proper roles. If you have deleted or changed the MID server keystore, and config.xml mid.instance.password value is encrypted, you may need to change this value to plain text (during MID startup, password is re-encrypted using current keystore and written back to mid.instance.password). at com.service_now.mid.services. StartupSequencer.runTests (StartupSequencer.java:664) at com.service_now.mid.services.StartupSequencer.startupSequencerRunnable(StartupSequencer.java:723) at java.base/java.lang.Thread.run(Thread.java:833) Note: The DEBUG lines would only be see if MID Server parameter mid.log.level=debug. In this situation, you will need to add that to the config.xml file directly, and then restart the MID Server. Changes in the instance won't sync down if it isn't talking to the instance in the first place.

Release Any instance version, but happening since November 2024.

Cause Since the Orlando release in 2019, MID Servers have checked that the instance certificate has not been revoked, using the OSCP Revocation lists of the certificate vendor.

At the time, all instances used a certificate from Entrust, which meant MID Servers needed access to the http://ocsp.entrust.net URL on port 80, as documented here , or the MID Servers would not connect. This caused hundreds of MID Server outages when the OCSP and Certificate Chain checks were added as part of the MID Security Policy feature, despite OCSP checks, and the need to access OCSP servers, being normal security best practice these days.

Starting November 2024, ServiceNow changed certificate vendor to DigiCert, which uses the OCSP URL http://ocsp.digicert.com instead . Customers were emailed at the time, and also a follow up communication will be (was?) sent in late January/Early February 2025.

This KB documented that change: KB1702083 Replacing Entrust Certificate Authority (CA)

Resolution All customers with hosted instances, that use MID Servers, will need to reconfigure the hosts and firewalls of MID Servers so that they can access DigiCert's OCSP URL , documented by DigiCert here: DigiCert Certificate Status IP Addresses

http://ocsp.digicert.com is the main one, but others are listed.

To test connectivity with the DigiCert OCSP server: Open a browser and go to ocsp.digicert.com/ping.html . You should receive the “You have successfully reached the DigiCert OCSP Service” message.

When testing from a browser, it is possible a different proxy server/firewall/web access gateway is used, compared to the MID Server Java application, which does not inherit proxy server settings from the host OS.

Note: This is on Port 80 . Why is explained in KB1709661 Why DigiCert OCSP and HTTP port 80 open on customer firewalls is required for MID Servers