MID Servers and Certificates - Support and Troubleshooting

MID Servers and Certificates Summary Table of Contents Introduction Summary of MID Server Certificate Checks Check introduction timeline 1. Outbound from MID Server to Instance, Integrations, or Proxy/Firewall in between 1.1 OCSP Check 1.2 Certificate Chain Check 1.3 Hostname Check 2. MID Server to Instance (for validation/encryption) 3. MID Server to Instance (for Authentication) 4. Inbound to MID Server Web Server extension from Integrations 4.1 SSL/TLS Connection 4.2 Key-Based Authentication 4.3 mTLS Authentication Introduction There is some confusion over the certificates and Java Keystores involved with MID Servers, which this KB hopes to clarify. It was first published by Tech Support in the Orlando/Paris timeframe, as undocumented changes were being made in patches and hotfixes. The Quebec documentation and requirements are now published by our developers, and so these Quebec pages should be referred to first, before getting too deep into the details mentioned below:-

MID Server certificate check policies This covers: TLS/SSL certificate validation Hostname validation Online Certificate Status Protocol (OCSP) MID Server security policy, including Certificate Chain MID Security Center KB0864770 MID Server certificate check security policies for self-hosted customers (also covers hosted instances where the URL is not ...service-now,com) KB0867397 MID Server TLS/SSL certificate check policy Quebec upgrade information KB0864766 MID Server connection failures when upgrading to Quebec (includes a temporary workaround for Quebec for all 3 checks, however this should not be used as a long term solution) Note: If Mutual Authentication is used, this workaround cannot be used. MID Server release notes (Quebec) Debugging Tip : Since Quebec, the debug logging for certificate related issues is much improved. Before getting too stuck into this, add this MID Server Parameter to config.xml and restart the MID Server:

Summary of MID Server Certificate Checks There are 3 main kinds of SSL/TLS certificate:

Trusted root CA signed, purchased from a vendor that is widely trusted already by Windows/Linux/Java. Internal CA signed, generated by the company active directory/domain server. By default, that CA won't be trusted. Self-signed, with no certificate chain or CA involved, usually generated by the endpoint/target device itself. By default, these are not trusted. Checked 3 ways:

Revocation check, in case the certificate was retired before the planned end date. Requires OCSP info inside the certificate, and access to the OCSP server URL provided. Certificate chain check, to ensure any intermediate and the root certificate are also valid. May require internal root CA certificates to be added to the 'cacerts' file. Hostname check, where the URL being access is compared with the Subject CN, or Subject Alternate Name values inside the cedrtificate for 3 different IP ranges:

'*.service-now.com' is the instance by default. Could be overridden for self-hosted on-premise/custom URL instances Intranet, for the private IP network ranges 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16 Internet, all other IPs

Check introduction timeline Version OCSP check Certificate Chain check Hostname check New York, and earlier N N N Orlando Y N N Orlando Patch 8 (and 7a) Y Y N Paris Y N N Paris Patch 2 (and 1 Hot Fix 3a, 1 Hot Fix 5a) Y Y N Quebec Y Y Y 1. Outbound from MID Server to Instance, Integrations, or Proxy/Firewall in between 1.1 OCSP Check Since Orlando, the MID Server needs to validate that certificates haven't been revoked for any reason, so the MID Server host needs to be able to connect to the OCSP /CRL servers of the certificate, as well as the instance URL.

As of the Paris release, that was purchased from Entrust, and so the URLs of the Entrust OCSP/CRL servers need to be open in your firewall. Entrust use DNS-based load balancing the IPs that these resolve may be different each time, so the URL always needs opening in the firewall. CRL: http://crl.entrust.net/level1k.crl OCSP: http://ocsp.entrust.net

You can use this tool on the 3rd party "SSL Labs" WebSite to see what the OCSP/CRL URLs are for the certificate of your instance. Swap the "hi.service-now.com" URL for your instance URL. The "Revocation information" section will give the URLs that will need opening in your firewall for the MID Server hosts. HTTP port 80 is normal for OCSP/CRL servers, and as it is a security protocol, should normally be possible to make an exception for in your company's security policies.

If the certificate truly has been revoked, this tool will also confirm it. In this situation it is right for the MID Server to block the endpoint:

MID Server Agent log will have this error for multiple threads:

WARNING *** WARNING *** OCSP revoke check IOException for *.service-now.com WARNING *** WARNING *** org.apache.commons.httpclient.HttpException: Connection reset Documentation links:

MID Server system requirements Wikipedia: Online Certificate Status Protocol (OCSP) MID Server certificate check policies (Quebec) Knowledge Base articles:

KB0854165 MID server OSCP requirement details (includes a temporary workaround for Orlando/Paris , however this should never be used as a long term solution) KB0864766 MID Server connection failures when upgrading to Quebec (includes a temporary workaround for Quebec for all 3 checks, however this should never be used as a long term solution) Related Problems:

PRB1385357 MID Server can fail to install or upgrade to Orlando due to new external connectivity requirement to ocsp.entrust.net for OCSP certification revocation verification check PRB1417416 Concurrency performance: JVM locks on OCSPCheckedCertificateCache are not released leading to import schedules taking exponentially longer (WARNING *** Freed stuck JVM lock: OCSPCheckedCertificateCache) - Quebec fix. PRB1419209 MID Server OCSP timeouts not throwing exceptions causing connection to instance to stay forever PRB1473617 1.2 Certificate Chain Check Keystore File: .\agent\jre\lib\security\ cacerts Default Password: changeit

These are Public keys for the SSL/TLS (Transport Layer Security) certificates of outgoing HTTPS connections. Outgoing integrations, such as to the Instance, LDAPS servers, or any proxy/firewall in between, will need to set up the secure HTTPS connection using a certificate. HTTP connections are forbidden for the connections to the ServiceNow instance. If the server we make the request to has a self-signed certificate, or one that does not have the normal root certificates in it's chain, then you will require the certificate loading into the cacerts file.

The best long term solution is to buy a certificate from a recognized Root CA for the network device. MID Servers will trust these automatically without needing to do anything.

This is done using the agent\jre\bin\ keytool.exe (/agent/jre/bin/keytool for linux). The whole chain of certificates may need adding, including intermediate certificates.

All required certificates for all instances and mid servers could be loaded into the same cacerts file, and then copy that file to all MID Servers. It's also a good idea to then keep a backup of that file.

This is not just used for 3rd party integrations. If a Proxy server, firewall, Load Balancer or other transparent network device is intercepting the traffic from MID Server to instance, then the certificate the MID Server needs to trust is the network devices, not the instance's. You can confirm this by opening the instance in a browser on the MID Server host and checking the Certificate. If it is the *.service-now.com one backed by Entrust, then nothing needs doing. If it is not, then you will possibly have to export that certificate from the browser , and all the others up the chain, and import those to the cacerts file.

In your browser accessing the instance from the MID Server host, you would expect the certificate chain to look like this (as of Paris timeframe). Click the padlock icon next to the url, and select certificate:

HTTPS Inspection / SSL Interception If the chain looks different, then you probably have a transparent network device in-between that is using the normal certificate between it and the instance, but substituting a different certificate for the HTTPS connection between the MID Server and itself. In the below examples, the blacked out parts were the customer's company name, suggesting these are self-signed certificates generated from their own CA server.

This will usually be due to a Web application firewall (WAF) implementing HTTPS/SSL Inspection/Interception, which is a form of Deep Packet Inspection (DPI) , and in effect a man-in-the-middle attack, in order to filter, monitor, and block anything malicious. The MID Server needs to be told it can trust the certificate issued by that device, to know it is authorized and not a malicious man-in-the-middle attack.

Some proxy/firewall/load balancer devices may be set to regularly change the certificate, and the certificate you manually added to cacerts no longer works. e.g. Zscaler load balancers have been seen to do this every 2 weeks. The only solution may be to add an exception/bypass to the device to turn of interception/inspection for mid server hosts connecting to instance URLs, or purchase a certificate from a public root CA that Java's default cacerts file already trusts.

If a traditional Proxy server is used, the hostname will be shown in the MID Server Configuration Parameters on the MID Server form. If you don't see the mid.proxy.use_proxy=true parameter, there may still be a proxy/firewall intercepting the connection.

For instances with worker nodes, and the MID Servers are set to connect to the special URL for worker nodes , the same certificate is also used, which is also used with the install.service-now.com server that has the install and upgrade files.

Orlando Patch 7a/8, Paris Patch 1 Hot Fix 3a/5a, Paris Patch 2, and Quebec add code to check certificate chains for all outbound requests using TLS certificates (PRB1419895), which may cause errors due to missing certificates in cacerts that had previously gone unchecked, breaking integrations and the connection to the instance . The above workaround for turning the OCSP check off temporarily also turns off this check.

Knowledge Base articles:

KB0816002 How to obtain SSL certificate from the browser KB0863109 MID Server cannot connect to ServiceNow instance, error in agent log: org.apache.commons.httpclient.HttpException: javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated KB0746078 Retiring TLS1.0 and 1.1 MID Servers and Worker Nodes KB0854165 MID server OSCP requirement details (includes a temporary workaround for Orlando/Paris , however this should not be used as a long term solution) KB0864766 MID Server connection failures when upgrading to Quebec (includes a temporary workaround for Quebec for all 3 checks, however this should not be used as a long term solution ) Documentation links:

Add SSL certificates for the MID Server Oracle Java: keytool Wikipedia: HTTPS , Transport Layer Security (TLS/SSL) , Java Keystore MID Server certificate check policies (Quebec) Related Problems:

PRB1419895 Lack of Certificate Validation KB0861038 Orlando Patch 7a / KB0861889 Paris Patch 1 Hotfix 3a Security fix Release Notes PRB1447511 Hotfix for PRB1419895 Lack of Certificate Validation causes MID Server DOWN . PRB1320637 A MID Server upgrade that includes a new JRE version will overwrite the cacerts file PRB1451866 The fix for PRB1320637 requires that the cacerts Truststore file password remains as the default "changeit", which many customers won't allow, causing certificate deletion during JRE upgrades (e.g. Quebec) and subsequent MID Server and Integration outage PRB1454071 MID Server TrustStore migrator will not migrate certificates flagged as CA-issued PRB1473617 Changing password of cacerts file prior to JRE upgrade causes error: Failed to load the jvm key store using the default password or specified password Examples The first 2 examples include " Request not sent to uri ", which means the MID Server decided it could not trust the URI and not send the request at all. This is a good clue that it is something about the endpoint certificates.

Connecting to the instance through a proxy/firewall may see this error in the agent logs. This happens to be from the JAR file requests, but all other requests to the instance will have a similar error. Note the "uri" in the error is for the instance, which is the diagnostic error for this cause. In this case the certificate required was for the network device URL, not the instance URL. " Unable to find certificate chain " is the clue here.

File sync worker: ecc_agent_jar WARNING *** WARNING *** Request not sent to uri = https:// .service-now.com/MIDFileSyncSnapshot.do?SOAP : javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed : java.security.cert.CertPathBuilderException: Unable to find certificate chain . File sync worker: ecc_agent_jar SEVERE *** ERROR *** SOAP Request: true ecc_agent_jar File sync worker: ecc_agent_jar SEVERE *** ERROR *** SOAP Response: Status code=0, Response body=null File sync worker: ecc_agent_jar WARNING *** WARNING *** Could not get file sync snapshot because: Request not sent to uri= https:// .service-now.com/MIDFileSyncSnapshot.do?SOAP : javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: java.security.cert.CertPathBuilderException: Unable to find certificate chain. A similar error for the instance certificate, where the Entrust root certificate was not available, and required adding to the cacerts file. This happens to be from the InstanceInfo request, but all other requests to the instance will have a similar error " peer not authenticated " is the clue here.

StartupSequencer WARNING *** WARNING *** Unable to get InstanceInfo: Request not sent to uri = https:// .service-now.com/InstanceInfo.do?SOAP : org.apache.commons.httpclient.HttpException: javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated (Network Configuration issue) Please check that the MID server can ping the instance: https://c .service-now.com/ You may also need to configure the network that the MID server uses to allow traffic over TCP port 443. An LDAPProbe import, with a missing certificate for the AD server, as seen from the Test Load records form in the instance, which will also be seen in the MID Server agent log. " No issuer certificate for certificate in certification path found " is the clue here.

MID Server reported error: javax.naming.CommunicationException: RTCDOMPRD03.rt.corp:636 [Root exception is javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: java.security.cert.CertPathBuilderException: No issuer certificate for certificate in certification path found .] at com.sun.jndi.ldap.Connection. (Connection.java:238) at com.sun.jndi.ldap.LdapClient. (LdapClient.java:137) at com.sun.jndi.ldap.LdapClient.getInstance(LdapClient.java:1609) at com.sun.jndi.ldap.LdapCtx.connect(LdapCtx.java:2749) at com.sun.jndi.ldap.LdapCtx. (LdapCtx.java:319) at com.sun.jndi.ldap.LdapCtxFactory.getUsingURL(LdapCtxFactory.java:192) at com.sun.jndi.ldap.LdapCtxFactory.getUsingURLs(LdapCtxFactory.java:210) at com.sun.jndi.ldap.LdapCtxFactory.getLdapCtxInstance(LdapCtxFactory.java:153) at com.sun.jndi.ldap.LdapCtxFactory.getInitialContext(LdapCtxFactory.java:83) at javax.naming.spi.NamingManager.getInitialContext(NamingManager.java:684) at javax.naming.InitialContext.getDefaultInitCtx(InitialContext.java:313) at javax.naming.InitialContext.init(InitialContext.java:244) at javax.naming.ldap.InitialLdapContext. (InitialLdapContext.java:154) at com.service_now.mid.probe.LDAPProbe.verifyConnectivity(LDAPProbe.java:124) at com.service_now.mid.probe.LDAPProbe.doWork(LDAPProbe.java:99) at com.service_now.mid.probe.LDAPProbe.probe(LDAPProbe.java:77) at com.service_now.mid.probe.AProbe.process(AProbe.java:103) A JavascriptProbe, with missing certificate for an endpoint:

Agent Log: Worker-Standard:JavascriptProbe-05422d31db9c9050d6f9f9b2f396193f WARNING *** WARNING *** javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: java.security.cert.CertPathBuilderException: Unable to find certificate chain . Caused by error in MID Server script include 'CSMIDServerRemoteFileImport' at line 23

Wrapper Log: Worker-Standard:JavascriptProbe-05422d31db9c9050d6f9f9b2f396193f, SEND TLSv1.2 ALERT: fatal, description = certificate_unknown Worker-Standard:JavascriptProbe-05422d31db9c9050d6f9f9b2f396193f, WRITE: TLSv1.2 Alert, length = 2 Worker-Standard:JavascriptProbe-05422d31db9c9050d6f9f9b2f396193f, called closeSocket() Worker-Standard:JavascriptProbe-05422d31db9c9050d6f9f9b2f396193f, handling exception: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: java.security.cert.CertPathBuilderException: Unable to find certificate chain. 1.3 Hostname Check New in Quebec.

Example errors - TBC.

Documentation links:

MID Server certificate check policies (Quebec) Knowledge Base articles:

KB0864766 MID Server connection failures when upgrading to Quebec (includes a temporary workaround for Quebec for all 3 checks, however this should not be used as a long term solution) 2. MID Server to Instance (for validation/encryption) Keystore File: .\agent\keystore\ agent_keystore.jks

This is a private key, automatically created on MID Server startup if it is not there, and contains a certificate to establish the trust between instance and mid server, and for decrypting data such as discovery credentials and other ecc_queue payload data that is encrypted. The MID Server cannot be validated to run any jobs unless that file is correct, or decrypt credentials from the instance to use when running jobs.

It is often recreated automatically on first startup after upgrade, and is also regularly refreshed.

When that refresh works, it looks something like this in the agent log of the MID Server:

StartupSequencer Updated public key, new certificate: [0] Version: 3 SerialNumber: 1603732292 IssuerDN: CN=snc-mid-datacenterdev_dva400_disco_midserver17-52721102f0331010e36955d33f0f9500,DC=service-now,DC=com Start Date: Mon Oct 26 10:11:32 PDT 2020 Final Date: Tue Oct 26 10:11:32 PDT 2021 SubjectDN: CN=snc-mid-datacenterdev_dva400_disco_midserver17-52721102f0331010e36955d33f0f9500,DC=service-now,DC=com Public Key: RSA Public Key [30:57:b5:a0:57:98:94:a2:e2:66:dd:79:b3:7f:37:6c:d8:9a:31:ff],[56:66:d1:a4] modulus: d2cd8c54a36990c57bd4146eea3677913b491fd82541ea462d15cf22a057a010c88bcd154f910134251ceccbbcbc5bbf00315199f557428986fc21b041fe4f8941ddc43a79d0257cc01c48fd681a39a7490255a6b2fb63dbaf01184727d15dcebfc9d1d2e3a532ece1512b418626f8c663d193d9cb896db29bec44fafcea63458b5497964155580d6389653a4613c4ceabd52d4a60228f127e59da4c32d394eab013ea0d3bb236c1ad1d5642135869de2047124164f7ea2f1a692df2e6bf174d896ea4c49c93e979cd905875c528061df8867fd06ac79f17b882ecaedf75bb54535a71d52368dff6a3baecc45e9ae9202289ccda7621015ff2372ac22906e0d1 public exponent: 10001 Signature Algorithm: SHA512WITHRSA Signature: 6a6202d16462659433f461c5153207a2aa6a63dc 5c4263c1ac2af3aba7da8b2b0208606db958ae5a 577f38df6a2fdfb80d21f5306a1ec0e695a64cf6 aa97b4da715f19d41907df95e4e6a2e4e6157577 9e68407d48a794248a76890e6bfc22cb60e29db2 7100976fd5d2f947aa267cda262e6b2ba52e705d af47ff7cb972b44b0238b5d0eda24a252499d304 8845e00bb112e762186755ce3079ec45194cfc0f 23b3adca0f7a38137aea35e1bf978f3f65b4cfe2 68aae654469716d54e2fda433b1142fe478fccb0 f1ae683e050ba14062e8593e4cc4e684c68f339e 09e376c3b8f74fc8d54bd4452dbc4a4cfbf9d047 a42cdad2e608eeab1ea16a753bb54923 When that fails, it may look more like:

StartupSequencer SEVERE *** ERROR *** UpdatePublicKey error: Digital signature of new public key must be provided. MID Server encryption keys do not match and are no longer valid. To restore proper functionality, invalidate and re-validate the MID Server. StartupSequencer WARNING *** WARNING *** Unable to update public key, will try again next MID Server restart Typically, it will say your signature is bad rather than missing. e.g.

SEVERE *** ERROR *** Unable to load keystore: Unexpected IOException loading KeyStore, caused by: Keystore was tampered with, or password was incorrect SEVERE *** ERROR *** Keystore and config.xml files out of sync. 1) Delete keystore/agent_keystore.jks or restore config.xml to its previous state, 2) ensure MID server has write permissions to config.xml and to keystore directory, 3) restart MID server. WARNING *** WARNING *** Encountered error: [Unable to load keystore] while starting up the service. Retry... If the MID Server remain UP, then clicking the Re-Key related link on the MID Server form can sometimes resolve this.

If the MID Server is Down, then deleting the file .\agent\keystore\agent_keystore.jks, clearing the keypairs.mid_id parameter value from the config.xml file, and restarting the MID Server service usually resolves this . A new good file should be created automatically on startup. The MID Server might still need validating from the MID Server record in the instance. Note: Since Quebec, this file may also include web server certificates, and mutual authentication certificates, and so those will need re-importing again afterwards.

Rekey a MID Server When a custom certificate is used, the Re-key and Validate UI-actions are disabled on the instance for the MID Server. A new UI action called Remove custom certificate is available to switch back to using a self-signed certificate. Using this action will cause the MID Server to remove the custom certificate and generate a new self-signed certificate, similar to the re-key option.

Docs: Unified Keystore (Quebec) The Rome release will move this Unified Keystore from agent/keystore/agent_keystore.jks to a new location agent/security/agent_keystore. If FIPS mode is enabled it also gets converted to BCFKS format.

3. MID Server to Instance (for Authentication) Keystore: Unified Keystore

From Quebec, a MID Server can optionally use Certificate-based Mutual Authentication (mTLS) to authenticate with the instance, instead of username/password basic authentication.

Docs: Unified Keystore (Quebec) Enable MID Server mutual authentication Quebec MID Server Release Notes New in the Quebec release Install a MID Server on Windows (Quebec) - See "Authentication Type" field. Known problems:

AMB Channel may fail to connect with certain certificates, where Subject CN is different to SAN DNS Name (problem TBC) Converting a MID server to mTLS after it is already Validated with with a username/password (problem TBC) Note: Up to at least Quebec, Mutual Authentication for outbound web services (anything other than the instance) is not supported by MID Server. e.g. REST Message to an endpoint straight from the instance can be done, when certificates are added to the instance, but not via a mid server.

4. Inbound to MID Server Web Server extension from Integrations 4.1 SSL/TLS Connection Keystore File: Unified Keystore (Quebec and later)

When the MID Server is acting as a Web Server for receiving push integrations, such as inbound REST/SOAP, Event Management event collection, or for Agent Client Collector installs, the certificate for this web server is an auto-generated self signed one by default. This is likely to cause problems for your security policies and need replacing with a trusted one, usually one generated by an internal CA, or a purchased one, especially if it is a internet-facing web server.

This will be a private key for the web server, supplied/generated by the customer.

This is added to the Unified Keystore (.\agent\securuty\keystore) using the Keytool utility. Only the certificate of the host needs adding, so the file has just 1 certificate.

This certificate is for the URL that will be used when sending requests to the MID Server's Web server, which may not always be the hostname of the mid server, if DNS aliases or a load balancer are used.

The Event Management feature has the best documentation on this (for Metrics collection from Agent Client Collector), although other features use the same web server extension integration feature:

Quebec and later version using Unified Keystore:

Configure the MID Web Server extension Event Management MID Web Server extension form Install custom certificates in the MID Server unified key store Note: In the Paris version and earlier, this was done differently, using a certificate added to .\agent\keystore\webserver_keystore.jceks. Ignore any documentation and KBs still mentioning that.

PRB1454515 Adding back the custom keystore ability on the MID web server (to support certificate chains for MID web servers) - Fixed in Rome. 4.2 Key-Based Authentication This is used by Agent Client Collector to authenticate with the MID Server.

Configure keybased MID Web Server authentication Note: In Rome, this limits to a single key per MID Server. The San Diego release allows for multiple keys at any one time, to help with key rotation.

4.3 mTLS Authentication From Rome Patch 4, Certificate-based Mutual Authentication is available:

MID Web Server mTLS Authentication