How to configure and troubleshoot MID Server certificates - Support and Troubleshooting

How to configure and troubleshoot MID Server certificates Summary .ns-kb-css-body-editor-container { p { font-size: 12pt; font-family: Lato; color: #000000; } span { font-size: 12pt; font-family: Lato; color: #000000; } h2 { font-size: 24pt; font-family: Lato; color: black; } h3 { font-size: 18pt; font-family: Lato; color: black; } h4 { font-size: 14pt; font-family: Lato; color: black; } a { font-size: 12pt; font-family: Lato; color: #00718F; } a:hover { font-size: 12pt; color: #024F69; } a:target { font-size: 12pt; color: #032D42; } a:visited { font-size: 12pt; color: #00718f; } ul { font-size: 12pt; font-family: Lato; } li { font-size: 12pt; font-family: Lato; } img { display: ; max-width: ; width: ; height: ; } } Learn how certificates work with MID Servers, including SSL/TLS validation, OCSP revocation checks, certificate chain verification, and hostname validation. This article covers outbound connections from the MID Server to instances and integrations, inbound connections to the MID Server Web Server extension, and troubleshooting common certificate errors.

Note : For the most current certificate requirements, see the product documentation on MID Server certificate check policies .

In this article MID Server certificate checks Certificate types and validation methods Outbound connections from MID Server MID Server to instance validation and encryption MID Server to instance authentication Inbound connections to MID Server Web Server extension MID Server certificate checks MID Servers perform three certificate validation checks for outbound connections:

OCSP revocation check Certificate chain check Hostname check Debugging tip: To enable detailed certificate logging, add this parameter to config.xml and restart the MID Server:

Certificate types and validation methods SSL/TLS certificate types Trusted root CA signed — Purchased from a vendor widely trusted by Windows, Linux, and Java Internal CA signed — Generated by company Active Directory or domain server; not trusted by default Self-signed — No certificate chain or CA involved; usually generated by the endpoint device; not trusted by default Validation methods Revocation check (OCSP) — Checks whether the certificate was revoked before its planned end date; requires OCSP information inside the certificate and access to the OCSP server URL Certificate chain check — Verifies that intermediate and root certificates are valid; may require adding internal root CA certificates to the cacerts file Hostname check — Compares the accessed URL with the Subject CN or Subject Alternate Name values in the certificate IP range categories Instance — *.service-now.com by default; can be overridden for self-hosted or on-premise/custom URL instances Intranet — Private IP ranges: 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16 Internet — All other IP addresses All three certificate validation checks (OCSP, certificate chain, and hostname) are enabled by default in all currently supported releases.

Outbound connections from MID Server OCSP check The MID Server validates that certificates have not been revoked. The MID Server host must be able to connect to the OCSP/CRL servers of the certificate in addition to the instance URL.

Firewall requirements ServiceNow instance certificates use Entrust. Open these URLs in your firewall for MID Server hosts:

CRL OCSP Note : Entrust uses DNS-based load balancing, so IP addresses may change. Configure your firewall to allow the URLs rather than specific IP addresses.

Identifying OCSP/CRL URLs for your instance Use the SSL Labs website to check your instance certificate. Replace the example URL with your instance URL and locate the Revocation information section to find the required URLs that need to be opened in your firewall for MID Server hosts. .

SSL Labs website tool

HTTP port 80 is standard for OCSP/CRL servers. Since OCSP/CRL is a security protocol, it should be possible to request a firewall exception in your company's security policies.

If the certificate has been revoked, this tool also confirms the revocation status. In this case, the MID Server correctly blocks the endpoint.

Error symptoms

The MID Server agent log shows this error for multiple threads:

WARNING *** WARNING *** OCSP revoke check IOException for *.service-now.com WARNING *** WARNING *** org.apache.commons.httpclient.HttpException: Connection reset

Certificate Chain Check Keystore location : .\agent\jre\lib\security\cacerts

Default password : changeit

The cacerts file contains public keys for SSL/TLS certificates used in outgoing HTTPS connections. Outgoing integrations to the instance, LDAPS servers, or any proxy or firewall in between require a secure HTTPS connection using a certificate.

If the server has a self-signed certificate or one without standard root certificates in its chain, you must add the certificate to the cacerts file.

Note: The best long-term solution is to purchase a certificate from a recognized root CA. MID Servers trust these automatically.

Adding certificates to cacerts Use the keytool utility to add certificates:

Windows : .\agent\jre\bin\keytool.exe Linux : ./agent/jre/bin/keytool You may need to add the entire certificate chain, including intermediate certificates. You can load all required certificates for all instances and MID Servers into a single cacerts file and copy it to all MID Servers. Keep a backup of this file.

Proxy and network device considerations

This certificate check applies to more than third-party integrations. If a proxy server, firewall, load balancer, or other network device intercepts traffic between the MID Server and instance, the MID Server must trust the network device certificate, not the instance certificate.

To verify which certificate is in use:

Open a browser on the MID Server host. Go to your instance URL. Select the padlock icon and view the certificate. If the certificate shows *.service-now.com backed by Entrust, no action is needed. If it shows a different certificate (typically containing your company name), export that certificate and all certificates in its chain, then import them to the cacerts file.

HTTPS Inspection / SSL Interception If the certificate chain looks different than expected, a network device may be implementing HTTPS/SSL inspection. This is a form of deep packet inspection where the device uses the normal certificate between itself and the instance but substitutes a different certificate for the connection between the MID Server and itself.

The MID Server must trust this certificate to confirm the interception is authorized and not a malicious attack.

Note: Some proxy, firewall, or load balancer devices regularly rotate certificates. If a manually added certificate stops working, you may need to add an exception to disable inspection for MID Server hosts connecting to instance URLs, or purchase a certificate from a public root CA that the default cacerts file already trusts.

If a traditional proxy server is used, the hostname appears in the MID Server Configuration Parameters on the MID Server form. If you don't see the mid.proxy.use_proxy=true parameter, a proxy or firewall may still be intercepting the connection.

For instances with worker nodes, and the MID Servers are set to connect to the special URL for worker nodes , the same certificate is also used, which is also used with the install.service-now.com server that has the install and upgrade files.

Error examples The first two examples include "Request not sent to uri," which means the MID Server determined it could not trust the URI and did not send the request. This is a key indicator that the issue involves endpoint certificates.

Missing certificate chain (proxy or firewall)

This error appears in the agent logs when connecting to the instance through a proxy or firewall. This example shows a JAR file request, but all other requests to the instance produce a similar error. Note that the "uri" in the error points to the instance, which helps diagnose this cause. In this case, the required certificate was for the network device URL, not the instance URL.

Clue: "Unable to find certificate chain"

File sync worker: ecc_agent_jar WARNING *** WARNING *** Request not sent to uri = https:// .service-now.com/MIDFileSyncSnapshot.do?SOAP : javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed : java.security.cert.CertPathBuilderException: Unable to find certificate chain . File sync worker: ecc_agent_jar SEVERE *** ERROR *** SOAP Request: true ecc_agent_jar File sync worker: ecc_agent_jar SEVERE *** ERROR *** SOAP Response: Status code=0, Response body=null File sync worker: ecc_agent_jar WARNING *** WARNING *** Could not get file sync snapshot because: Request not sent to uri= https:// .service-now.com/MIDFileSyncSnapshot.do?SOAP : javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: java.security.cert.CertPathBuilderException: Unable to find certificate chain. Missing Entrust root certificate

This error occurs when the Entrust root certificate is not available and must be added to the cacerts file. This example shows an InstanceInfo request, but all other requests to the instance produce a similar error.

Clue: "peer not authenticated"

StartupSequencer WARNING *** WARNING *** Unable to get InstanceInfo: Request not sent to uri = https:// .service-now.com/InstanceInfo.do?SOAP : org.apache.commons.httpclient.HttpException: javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated (Network Configuration issue) Please check that the MID server can ping the instance: https://c .service-now.com/ You may also need to configure the network that the MID server uses to allow traffic over TCP port 443. Missing LDAP server certificate

This error occurs during an LDAPProbe import when the certificate for the Active Directory server is missing. This error appears in the Test Load records form in the instance and in the MID Server agent log.

Clue: "No issuer certificate for certificate in certification path found"

MID Server reported error: javax.naming.CommunicationException: RTCDOMPRD03.rt.corp:636 [Root exception is javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: java.security.cert.CertPathBuilderException: No issuer certificate for certificate in certification path found .] at com.sun.jndi.ldap.Connection. (Connection.java:238) at com.sun.jndi.ldap.LdapClient. (LdapClient.java:137) at com.sun.jndi.ldap.LdapClient.getInstance(LdapClient.java:1609) at com.sun.jndi.ldap.LdapCtx.connect(LdapCtx.java:2749) at com.sun.jndi.ldap.LdapCtx. (LdapCtx.java:319) at com.sun.jndi.ldap.LdapCtxFactory.getUsingURL(LdapCtxFactory.java:192) at com.sun.jndi.ldap.LdapCtxFactory.getUsingURLs(LdapCtxFactory.java:210) at com.sun.jndi.ldap.LdapCtxFactory.getLdapCtxInstance(LdapCtxFactory.java:153) at com.sun.jndi.ldap.LdapCtxFactory.getInitialContext(LdapCtxFactory.java:83) at javax.naming.spi.NamingManager.getInitialContext(NamingManager.java:684) at javax.naming.InitialContext.getDefaultInitCtx(InitialContext.java:313) at javax.naming.InitialContext.init(InitialContext.java:244) at javax.naming.ldap.InitialLdapContext. (InitialLdapContext.java:154) at com.service_now.mid.probe.LDAPProbe.verifyConnectivity(LDAPProbe.java:124) at com.service_now.mid.probe.LDAPProbe.doWork(LDAPProbe.java:99) at com.service_now.mid.probe.LDAPProbe.probe(LDAPProbe.java:77) at com.service_now.mid.probe.AProbe.process(AProbe.java:103) JavascriptProbe with missing endpoint certificate:

This error occurs when a JavascriptProbe cannot verify the certificate for an endpoint.

Clue: "Unable to find certificate chain"

Agent Log: Worker-Standard:JavascriptProbe-05422d31db9c9050d6f9f9b2f396193f WARNING *** WARNING *** javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: java.security.cert.CertPathBuilderException: Unable to find certificate chain . Caused by error in MID Server script include 'CSMIDServerRemoteFileImport' at line 23

Wrapper Log: Worker-Standard:JavascriptProbe-05422d31db9c9050d6f9f9b2f396193f, SEND TLSv1.2 ALERT: fatal, description = certificate_unknown Worker-Standard:JavascriptProbe-05422d31db9c9050d6f9f9b2f396193f, WRITE: TLSv1.2 Alert, length = 2 Worker-Standard:JavascriptProbe-05422d31db9c9050d6f9f9b2f396193f, called closeSocket() Worker-Standard:JavascriptProbe-05422d31db9c9050d6f9f9b2f396193f, handling exception: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: java.security.cert.CertPathBuilderException: Unable to find certificate chain. Key error indicators Use these phrases to quickly identify certificate issues in log files:

Error phrase Indicates "Unable to find certificate chain" Missing intermediate or root certificate "peer not authenticated" Certificate not trusted "No issuer certificate for certificate in certification path found" Missing issuer certificate in chain "Request not sent to uri" MID Server rejected the connection before sending

Hostname check The MID Server validates that the hostname in the URL matches the Subject CN or Subject Alternate Name in the certificate.

For configuration details, see MID Server certificate check policies in product documentation.

MID Server to instance validation and encryption Keystore location : .\agent\keystore\agent_keystore.jks

This keystore contains a private key that establishes trust between the instance and MID Server automatically upon startup. It is used for decrypting data such as Discovery credentials and other ecc_queue payload data. The MID Server cannot run jobs or decrypt credentials from the instance without a valid keystore file.

The keystore is created automatically on MID Server startup if it does not exist and is regularly refreshed.

Successful key refresh

When the key refresh succeeds, the agent log shows output similar to:

StartupSequencer Updated public key, new certificate: [0] Version: 3 SerialNumber: 1603732292 IssuerDN: CN=snc-mid-datacenterdev_dva400_disco_midserver17-52721102f0331010e36955d33f0f9500,DC=service-now,DC=com Start Date: Mon Oct 26 10:11:32 PDT 2020 Final Date: Tue Oct 26 10:11:32 PDT 2021 SubjectDN: CN=snc-mid-datacenterdev_dva400_disco_midserver17-52721102f0331010e36955d33f0f9500,DC=service-now,DC=com Public Key: RSA Public Key [30:57:b5:a0:57:98:94:a2:e2:66:dd:79:b3:7f:37:6c:d8:9a:31:ff],[56:66:d1:a4] modulus: d2cd8c54a36990c57bd4146eea3677913b491fd82541ea462d15cf22a057a010c88bcd154f910134251ceccbbcbc5bbf00315199f557428986fc21b041fe4f8941ddc43a79d0257cc01c48fd681a39a7490255a6b2fb63dbaf01184727d15dcebfc9d1d2e3a532ece1512b418626f8c663d193d9cb896db29bec44fafcea63458b5497964155580d6389653a4613c4ceabd52d4a60228f127e59da4c32d394eab013ea0d3bb236c1ad1d5642135869de2047124164f7ea2f1a692df2e6bf174d896ea4c49c93e979cd905875c528061df8867fd06ac79f17b882ecaedf75bb54535a71d52368dff6a3baecc45e9ae9202289ccda7621015ff2372ac22906e0d1 public exponent: 10001 Signature Algorithm: SHA512WITHRSA Signature: 6a6202d16462659433f461c5153207a2aa6a63dc 5c4263c1ac2af3aba7da8b2b0208606db958ae5a 577f38df6a2fdfb80d21f5306a1ec0e695a64cf6 aa97b4da715f19d41907df95e4e6a2e4e6157577 9e68407d48a794248a76890e6bfc22cb60e29db2 7100976fd5d2f947aa267cda262e6b2ba52e705d af47ff7cb972b44b0238b5d0eda24a252499d304 8845e00bb112e762186755ce3079ec45194cfc0f 23b3adca0f7a38137aea35e1bf978f3f65b4cfe2 68aae654469716d54e2fda433b1142fe478fccb0 f1ae683e050ba14062e8593e4cc4e684c68f339e 09e376c3b8f74fc8d54bd4452dbc4a4cfbf9d047 a42cdad2e608eeab1ea16a753bb54923 Key refresh errors

When the refresh fails, the agent log may show:

StartupSequencer SEVERE *** ERROR *** UpdatePublicKey error: Digital signature of new public key must be provided. MID Server encryption keys do not match and are no longer valid. To restore proper functionality, invalidate and re-validate the MID Server. StartupSequencer WARNING *** WARNING *** Unable to update public key, will try again next MID Server restart Or:

SEVERE *** ERROR *** Unable to load keystore: Unexpected IOException loading KeyStore, caused by: Keystore was tampered with, or password was incorrect SEVERE *** ERROR *** Keystore and config.xml files out of sync. 1) Delete keystore/agent_keystore.jks or restore config.xml to its previous state, 2) ensure MID server has write permissions to config.xml and to keystore directory, 3) restart MID server. WARNING *** WARNING *** Encountered error: [Unable to load keystore] while starting up the service. Retry... Resolving keystore errors

If the MID Server is UP, select Re-Key on the MID Server form in the instance.

If the MID Server is DOWN:

Delete the keystore file. ( .\agent\keystore\agent_keystore.jks) In config.xml, clear the value for the keypairs.mid_id parameter. Restart the MID Server service. Validate the MID Server from the MID Server record in the instance. A new keystore file is created automatically on startup.

Note: This file may also include web server certificates and mutual authentication certificates. If you delete the keystore, you must re-import those certificates.

Custom certificates

When a custom certificate is used, the Re-key and Validate actions are disabled on the MID Server form. Use the Remove custom certificate action to switch back to a self-signed certificate. This action removes the custom certificate and generates a new self-signed certificate.

MID Server to instance authentication Keystore: Unified Keystore

MID Servers can optionally use certificate-based mutual authentication (mTLS) instead of username and password basic authentication.

Known limitations:

The AMB Channel may fail to connect with certain certificates where the Subject CN differs from the SAN DNS Name. Converting a MID Server to mTLS after it is validated with username and password may cause issues. Mutual authentication for outbound web services (other than the instance) is not supported by MID Server. REST messages to an endpoint can use mTLS directly from the instance when certificates are added, but not through a MID Server.

Inbound connections to MID Server Web Server extension SSL/TLS connection Keystore: Unified Keystore

Location: .\agent\security\keystore

When the MID Server acts as a web server for receiving push integrations (such as inbound REST/SOAP, Event Management event collection, or Agent Client Collector installations), it uses an auto-generated self-signed certificate by default.

This default certificate may not meet your security policies. Replace it with a trusted certificate, typically one generated by an internal CA or purchased from a public CA, especially for internet-facing web servers.

Adding a custom certificate

Use the keytool utility to add your certificate to the unified keystore (.\agent\security\keystore). Add only the host certificate (the file contains one certificate). The certificate must match the URL used when sending requests to the MID Server web server, which may differ from the MID Server hostname if DNS aliases or a load balancer are used.

For configuration instructions, see the following in product documentation:

Configure the MID Web Server extension Install custom certificates in the MID Server unified key store

Key-Based Authentication Agent Client Collector uses key-based authentication to authenticate with the MID Server.

For configuration instructions, see Configure keybased MID Web Server authentication

mTLS Authentication Certificate-based mutual authentication (mTLS) is available for inbound connections to the MID Server Web Server extension.

For configuration instructions, see MID Web Server mTLS Authentication.

Release .ns-kb-css-body-editor-container { p { font-size: 12pt; font-family: Lato; color: #000000; } span { font-size: 12pt; font-family: Lato; color: #000000; } h2 { font-size: 24pt; font-family: Lato; color: black; } h3 { font-size: 18pt; font-family: Lato; color: black; } h4 { font-size: 14pt; font-family: Lato; color: black; } a { font-size: 12pt; font-family: Lato; color: #00718F; } a:hover { font-size: 12pt; color: #024F69; } a:target { font-size: 12pt; color: #032D42; } a:visited { font-size: 12pt; color: #00718f; } ul { font-size: 12pt; font-family: Lato; } li { font-size: 12pt; font-family: Lato; } img { display: ; max-width: ; width: ; height: ; } } All supported releases

Related Links .ns-kb-css-body-editor-container { p { font-size: 12pt; font-family: Lato; color: #000000; } span { font-size: 12pt; font-family: Lato; color: #000000; } h2 { font-size: 24pt; font-family: Lato; color: black; } h3 { font-size: 18pt; font-family: Lato; color: black; } h4 { font-size: 14pt; font-family: Lato; color: black; } a { font-size: 12pt; font-family: Lato; color: #00718F; } a:hover { font-size: 12pt; color: #024F69; } a:target { font-size: 12pt; color: #032D42; } a:visited { font-size: 12pt; color: #00718f; } ul { font-size: 12pt; font-family: Lato; } li { font-size: 12pt; font-family: Lato; } img { display: ; max-width: ; width: ; height: ; } } General MID Server certificate information MID Server certificate check policies Security Center release notes MID Server system requirements Outbound connections (OCSP, certificate chain, hostname checks) Add SSL certificates for the MID Server How to obtain SSL certificate from the browser MID Server cannot connect to instance due to SSL certificate "peer not authenticated" error MID Server connection failures after instance upgrade Retiring TLS 1.0 and 1.1 MID Servers and Worker Nodes (product documentation) MID Server to instance validation and encryption MID Server unified keystore Rekey a MID Server MID Server to instance authentication Enable MID Server mutual authentication Inbound connections to MID Server Web Server extension Configure the MID Web Server extension Event Management MID Web Server extension form Install custom certificates in the MID Server unified keystore Configure key-based MID Web Server authentication MID Web Server mTLS authentication