There is some confusion over the certificates and Java Keystores involved with MID Servers, which this KB hopes to clarify. It was first published by Tech Support in the Orlando/Paris timeframe, as undocumented changes were being made in patches and hotfixes. The Quebec documentation and requirements are now published by our developers, and so these Quebec pages should be referred to first, before getting too deep into the details mentioned below:-
Debugging Tip: Since Quebec, the debug logging for certificate related issues is much improved. Before getting too stuck into this, add this MID Server Parameter to config.xml and restart the MID Server:
<parameter name="mid.log.level" value="debug"/>
|Version||OCSP check||Certificate Chain check||Hostname check|
|New York, and earlier||N||N||N|
|Orlando Patch 8 (and 7a)||Y||Y||N|
|Paris Patch 2 (and 1 Hot Fix 3a, 1 Hot Fix 5a)||Y||Y||N|
Since Orlando, the MID Server needs to validate that certificates haven't been revoked for any reason, so the MID Server host needs to be able to connect to the OCSP/CRL servers of the certificate, as well as the instance URL.
As of the Paris release, that was purchased from Entrust, and so the URLs of the Entrust OCSP/CRL servers need to be open in your firewall. Entrust use DNS-based load balancing the IPs that these resolve may be different each time, so the URL always needs opening in the firewall.
You can use this tool on the 3rd party "SSL Labs" WebSite to see what the OCSP/CRL URLs are for the certificate of your instance. Swap the "hi.service-now.com" URL for your instance URL. The "Revocation information" section will give the URLs that will need opening in your firewall for the MID Server hosts. HTTP port 80 is normal for OCSP/CRL servers, and as it is a security protocol, should normally be possible to make an exception for in your company's security policies.
If the certificate truly has been revoked, this tool will also confirm it. In this situation it is right for the MID Server to block the endpoint:
MID Server Agent log will have this error for multiple threads:
WARNING *** WARNING *** OCSP revoke check IOException for *.service-now.com
WARNING *** WARNING *** org.apache.commons.httpclient.HttpException: Connection reset
Knowledge Base articles:
Keystore File: .\agent\jre\lib\security\cacerts
Default Password: changeit
These are Public keys for the SSL/TLS (Transport Layer Security) certificates of outgoing HTTPS connections. Outgoing integrations, such as to the Instance, LDAPS servers, or any proxy/firewall in between, will need to set up the secure HTTPS connection using a certificate. HTTP connections are forbidden for the connections to the ServiceNow instance. If the server we make the request to has a self-signed certificate, or one that does not have the normal root certificates in it's chain, then you will require the certificate loading into the cacerts file.
The best long term solution is to buy a certificate from a recognized Root CA for the network device. MID Servers will trust these automatically without needing to do anything.
This is done using the agent\jre\bin\keytool.exe (/agent/jre/bin/keytool for linux). The whole chain of certificates may need adding, including intermediate certificates.
All required certificates for all instances and mid servers could be loaded into the same cacerts file, and then copy that file to all MID Servers. It's also a good idea to then keep a backup of that file.
This is not just used for 3rd party integrations. If a Proxy server, firewall, Load Balancer or other transparent network device is intercepting the traffic from MID Server to instance, then the certificate the MID Server needs to trust is the network devices, not the instance's. You can confirm this by opening the instance in a browser on the MID Server host and checking the Certificate. If it is the *.service-now.com one backed by Entrust, then nothing needs doing. If it is not, then you will possibly have to export that certificate from the browser, and all the others up the chain, and import those to the cacerts file.
In your browser accessing the instance from the MID Server host, you would expect the certificate chain to look like this (as of Paris timeframe). Click the padlock icon next to the url, and select certificate:
If the chain looks different, then you probably have a transparent network device in-between that is using the normal certificate between it and the instance, but substituting a different certificate for the HTTPS connection between the MID Server and itself. In the below examples, the blacked out parts were the customer's company name, suggesting these are self-signed certificates generated from their own CA server.
This will usually be due to a Web application firewall (WAF) implementing HTTPS/SSL Inspection/Interception, which is a form of Deep Packet Inspection (DPI), and in effect a man-in-the-middle attack, in order to filter, monitor, and block anything malicious. The MID Server needs to be told it can trust the certificate issued by that device, to know it is authorized and not a malicious man-in-the-middle attack.
Some proxy/firewall/load balancer devices may be set to regularly change the certificate, and the certificate you manually added to cacerts no longer works. e.g. Zscaler load balancers have been seen to do this every 2 weeks. The only solution may be to add an exception to the device to turn of interception/inspection for mid server hosts connecting to instance URLs, or buy a proper certificate from a public root CA.
If a traditional Proxy server is used, the hostname will be shown in the MID Server Configuration Parameters on the MID Server form. If you don't see the mid.proxy.use_proxy=true parameter, there may still be a proxy/firewall intercepting the connection.
For instances with worker nodes, and the MID Servers are set to connect to the special URL for worker nodes, the same certificate is also used, which is also used with the install.service-now.com server that has the install and upgrade files.
Orlando Patch 7a/8, Paris Patch 1 Hot Fix 3a/5a, Paris Patch 2, and Quebec add code to check certificate chains for all outbound requests using TLS certificates (PRB1419895), which may cause errors due to missing certificates in cacerts that had previously gone unchecked, breaking integrations and the connection to the instance. The above workaround for turning the OCSP check off temporarily also turns off this check.
Knowledge Base articles:
The first 2 examples include "Request not sent to uri", which means the MID Server decided it could not trust the URI and not send the request at all. This is a good clue that it is something about the endpoint certificates.
Connecting to the instance through a proxy/firewall may see this error in the agent logs. This happens to be from the JAR file requests, but all other requests to the instance will have a similar error. Note the "uri" in the error is for the instance, which is the diagnostic error for this cause. In this case the certificate required was for the network device URL, not the instance URL. "Unable to find certificate chain" is the clue here.
File sync worker: ecc_agent_jar WARNING *** WARNING *** Request not sent to uri= https://<INSTANCE_NAME>.service-now.com/MIDFileSyncSnapshot.do?SOAP : javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: java.security.cert.CertPathBuilderException: Unable to find certificate chain.
File sync worker: ecc_agent_jar SEVERE *** ERROR *** SOAP Request: <SOAP-ENV:Envelope xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:SOAP-ENC="http://schemas.xmlsoap.org/soap/encoding/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:tns="http://www.service-now.com/GetMIDInfo" xmlns:m="http://www.service-now.com" xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/" SOAP-ENV:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/"><SOAP-ENV:Body><m:execute><attachments xsi:type="xsd:string">true</attachments><type xsi:type="xsd:string">ecc_agent_jar</type></m:execute></SOAP-ENV:Body></SOAP-ENV:Envelope>
File sync worker: ecc_agent_jar SEVERE *** ERROR *** SOAP Response: Status code=0, Response body=null
File sync worker: ecc_agent_jar WARNING *** WARNING *** Could not get file sync snapshot because: Request not sent to uri= https://<INSTANCE_NAME>.service-now.com/MIDFileSyncSnapshot.do?SOAP : javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: java.security.cert.CertPathBuilderException: Unable to find certificate chain.
A similar error for the instance certificate, where the Entrust root certificate was not available, and required adding to the cacerts file. This happens to be from the InstanceInfo request, but all other requests to the instance will have a similar error "peer not authenticated" is the clue here.
StartupSequencer WARNING *** WARNING *** Unable to get InstanceInfo: Request not sent to uri= https://<INSTANCE_NAME>.service-now.com/InstanceInfo.do?SOAP : org.apache.commons.httpclient.HttpException: javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated
(Network Configuration issue) Please check that the MID server can ping the instance: https://c<INSTANCE_NAME>.service-now.com/
You may also need to configure the network that the MID server uses to allow traffic over TCP port 443.
An LDAPProbe import, with a missing certificate for the AD server, as seen from the Test Load records form in the instance, which will also be seen in the MID Server agent log. "No issuer certificate for certificate in certification path found" is the clue here.
MID Server reported error: javax.naming.CommunicationException: RTCDOMPRD03.rt.corp:636 [Root exception is javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: java.security.cert.CertPathBuilderException: No issuer certificate for certificate in certification path found.]
Caused by error in MID Server script include 'CSMIDServerRemoteFileImport' at line 23
New in Quebec.
Example errors - TBC.
Knowledge Base articles:
Keystore File: .\agent\keystore\agent_keystore.jks
This is a private key, automatically created on MID Server startup if it is not there, and contains a certificate to establish the trust between instance and mid server, and for decrypting data such as discovery credentials and other ecc_queue payload data that is encrypted. The MID Server cannot be validated to run any jobs unless that file is correct, or decrypt credentials from the instance to use when running jobs.
It is often recreated automatically on first startup after upgrade, and is also regularly refreshed.
When that refresh works, it looks something like this in the agent log of the MID Server:
StartupSequencer Updated public key, new certificate:
 Version: 3
Start Date: Mon Oct 26 10:11:32 PDT 2020
Final Date: Tue Oct 26 10:11:32 PDT 2021
Public Key: RSA Public Key [30:57:b5:a0:57:98:94:a2:e2:66:dd:79:b3:7f:37:6c:d8:9a:31:ff],[56:66:d1:a4]
public exponent: 10001
Signature Algorithm: SHA512WITHRSA
When that fails, it may look more like:
StartupSequencer SEVERE *** ERROR *** UpdatePublicKey error: Digital signature of new public key must be provided. MID Server encryption keys do not match and are no longer valid. To restore proper functionality, invalidate and re-validate the MID Server.
StartupSequencer WARNING *** WARNING *** Unable to update public key, will try again next MID Server restart
Typically, it will say your signature is bad rather than missing. e.g.
SEVERE *** ERROR *** Unable to load keystore: Unexpected IOException loading KeyStore, caused by: Keystore was tampered with, or password was incorrect SEVERE *** ERROR *** Keystore and config.xml files out of sync. 1) Delete keystore/agent_keystore.jks or restore config.xml to its previous state, 2) ensure MID server has write permissions to config.xml and to keystore directory, 3) restart MID server. WARNING *** WARNING *** Encountered error: [Unable to load keystore] while starting up the service. Retry...
If the MID Server remain UP, then clicking the Re-Key related link on the MID Server form can sometimes resolve this.
If the MID Server is Down, then deleting the file .\agent\keystore\agent_keystore.jks, clearing the keypairs.mid_id parameter value from the config.xml file, and restarting the MID Server service usually resolves this. A new good file should be created automatically on startup. The MID Server might still need validating from the MID Server record in the instance.
Note: Since Quebec, this file may also include web server certificates, and mutual authentication certificates, and so those will need re-importing again afterwards.
When a custom certificate is used, the Re-key and Validate UI-actions are disabled on the instance for the MID Server. A new UI action called Remove custom certificate is available to switch back to using a self-signed certificate. Using this action will cause the MID Server to remove the custom certificate and generate a new self-signed certificate, similar to the re-key option.
The Rome release will move this Unified Keystore from agent/keystore/agent_keystore.jks to a new location agent/security/agent_keystore.
If FIPS mode is enabled it also gets converted to BCFKS format.
Keystore: Unified Keystore
From Quebec, a MID Server can optionally use Mutual Authentication (mTLS) to authenticate with the instance, instead of username/password basic authentication.
Up to at least Quebec, Mutual Authentication for outbound web services (anything other than the instance) is not supported by MID Server. e.g. REST Message to an endpoint straight from the instance can be done, when certificates are added to the instance, but not via a mid server.
Note: The following applies to Orlando/Paris, and might have changed in Quebec in relation to the Unified Keystore and Mutual Authentication/mTLS changes - TBC.
Keystore File: .\agent\keystore\webserver_keystore.jceks (<=Paris), or Unified Keystore (Quebec)
Password: Whatever you set it to be on manual creation
When the MID Server is acting as a Web Server for receiving push integrations, such as inbound REST/SOAP, Event Management event collection, or for Agent Client Collector installs, the certificate to use for this web server is uploaded to .\agent\keystore\webserver_keystore.jceks
This is the private key for the web server, supplied/generated by the customer.
This is also added using the Keytool utility. Only the certificate of the host needs adding, so the file has just 1 certificate.
This certificate is for the URL that will be used when sending requests to the MID Server's Web server, which may not always be the hostname of the mid server, if DNS aliases or a load balancer are used.
The Event Management feature has the best documentation on this (for Metrics collection from Agent Client Collector), although other features use the same web server extension integration feature:
Quebec version using Unified Keystore:
Paris version and earlier:
Rome version may have the files in a different place (agent/security instead of agent/keystore), but will probably still use the same unified keystore system in general - TBC
Error: java.lang.IllegalStateException: KeyStores with multiple certificates are not supported on the base class org.eclipse.jetty.util.ssl.SslContextFactory