Concurrency performance: JVM locks on OCSPCheckedCertificateCache are not released leading to import schedules taking exponentially longer (WARNING *** Freed stuck JVM lock: OCSPCheckedCertificateCache)


Description

JVM locks on OCSPCheckedCertificateCache are not being released on time when servicing concurrent requests.

The MID Server agent log will show errors like:

glide.lock.cleaner WARNING *** WARNING *** Freed stuck JVM lock: OCSPCheckedCertificateCache
Worker-Standard:JavascriptProbe-9cd773b24fe8a600f0c92ae6f110c720 [0:03:20.899] GlideSSLProtocolSocketFactory.walkCertChainToCheckIfRevoked of: CN=*.service-now.com, O="ServiceNow, Inc.", L=San Diego, ST=California, C=US

During high load operations, such as scheduled imports of data from customer's network to the ServiceNow instance using the MID Server, customers may notice symptoms of deterioration on their previously running schedules, that are taking exponentially longer time to complete, from few minutes in normal conditions to almost a day.

Steps to Reproduce

  1. Download and install mid server on an Orlando or above instance.
  2. Run an import operation on demand, or create a schedule and run.

Workaround

This problem is fixed in the next upcoming release. If you are able to upgrade, review the Fixed In section to determine the latest version with a permanent fix your instance can be upgraded to.

Possible workarounds are the following:

1) If you see HTTP connection resets in the logs

Resolution:

  1. In Windows based environments, if the domain controller is behind a firewall, you may have to configure the firewall to explicitly allow outgoing HTTP connections to enable the domain controller to connect to the OCSP responder.
  2. Verify if outgoing traffic to the Entrust OCSP server, from the MID Server host, is blocked by a firewall appliance or a proxy configuration. In this case, you will have to include in the allow-list the FQDNs of the Entrust responders as required by Entrust. (e.g. crl.entrust.net, crl2.entrust.net, ocsp.entrust.net, although this may change)
    If you are not using the ServiceNow certificate because you are using a vanity URL or your own certificate, then you can confirm the OCSP and CRL URLs using this 3rd party tool, replacing hi.service-now with your instance URL: https://www.ssllabs.com/ssltest/analyze.html?d=hi.service-now.com
  3. Clear the validation endpoints LIST by setting the value of MID Server property of mid.security.validation.endpoints to blank. This will turn off the external endpoint validation. Do not remove the property.

2) If you do not see a HTTP connection reset error

Clear the validation endpoints list by setting the value of MID Server property of mid.security.validation.endpoints to blank/empty. This will turn off the external endpoint validation. Do not remove the property.


Related Problem: PRB1417416