MID Server Mutual Authentication - Support and Troubleshooting

MID Server Mutual Authentication Description This Knowledge Base article covers the main steps involved in setting up MID server mutual authentication. This Knowledge Base article focuses solely on mutual authentication between the MID server and the instance.

Table of Contents Description Setup Files Troubleshooting Instance Troubleshooting Split a PEM File Errors Q&A Additional Information Setup Review the steps to setup Certificate-based authentication on your instance Certificate-based authentication Get the certificates from a CA Note: Your security team or certificate provider should be familiar on how to create root, intermediate, and client certificates. No information from Servicenow, or the instance, is required to create such certificates. Split up the PEM certificate into 3 different PEM files. There should be the following files: Private key and leaf certificate Leaf certificate Intermediate certificates + Root certificate On the instance, upload file 3 to the CA Certificate Chain table (sys_ca_certificate) Notes: If uploading individually instead of a bundle, start with the root certificate, and then the intermediate certificate(s) It is preferred to uploaded file 3 as a PEM bundle (set type to "CA Cert" in this case), instead of split into multiple files Wait for the uploaded CA Cert record on the instance to have the "Publish Status" transition to "Exists." Note: Do not proceed until the publish status is updated to "Exists". Attach the leaf certificate only to sys_user_certificate, no need for the private key. If the MID server is validated, invalidate the MID server before continuing If the MID server is running, stop the MID server On the MID server add the private key + certificate following the steps from: Enable MID Server mutual authentication Start the MID server Troubleshooting Note: Some of the following steps may require a support case so that our System Administrators may enable features

Validate the instance is setup with ADCv2 load balancers To validate, check the "Server" header on a request sent to the instance by issuing the command: curl -I https:// .service-now.com If Server header has a value of "snow_adc", the instance is on ADCv2 If there is any other values, the instance is not on ADCv2 Is the instance setup with ADCv2 load balancers? Yes: Continue No: Request migration to ADCv2 load balancers Validate the load balancer is setup for mutual authentication To validate, visit the URL: /adcv2/supports_tls If it returns mixed or true , the instance supports inbound TLS If the page is not found or is redirected, the instance does not support inbound TLS Is the load balancer setup for mutual authentication? Yes: Continue No: Request load balancer be configured for mutual authentication Validate mutual authentication plugin is enabled Navigate to: .service-now.com/$allappsmgmt.do?sysparm_redirect=true&sysparm_search=com.glide.auth.mutual">https:// .service-now.com/$allappsmgmt.do?sysparm_redirect=true&sysparm_search=com.glide.auth.mutual Confirm the plugin is "Installed" Is mutual authentication plugin enabled? Yes: Continue No: Enable plugin or request plugin activation Are there any errors following the documentation to setup sys_ca_certificate or sys_user_certificate? Yes: Are certificates in a supported format? (PEM) Yes:Continue No: Convert certificate to PEM format as advised by the certificate vendor No: Continue Are the sys_ca_certificate and sys_user_certificate active? Yes: Continue No: Set the field active = true Are there any errors following the documentation to setup the MID Server for mutual authentication? Yes: Resolve errors No: Continue Setup MID Server SSL debug Set MID Server SSL Debug If necessary, setup debug parameters on the instance (see "Instance Troubleshooting" section of this KB) Reproduce the issue Review MID Server wrapper file for any handshake errors, review the steps in a handshake in following link Client-authenticated TLS handshake Instance Troubleshooting Note: To avoid any issue, do not change following properties/scripts without consulting technical support

Support may need to set following properties to true:

glide.auth.debug.enabled glide.auth.ssl.debug.enabled Support may also need to update script include MutualAuth (sys_script_include.do?sys_id=94d0d81373211010978acff8faf6a7c9) like so:

getAuthorized : function() { var request = GlideTransaction.get().getRequest(); var certHeader = request.getHeader("X-Client-Cert"); gs.log("MutualAuth certHeader===>" + certHeader); ... ... } Split a PEM File PEM files are just a collection of keys/certificates inside BEGIN / END blocks like the following:

-----BEGIN ENCRYPTED PRIVATE KEY----- -----END ENCRYPTED PRIVATE KEY----- You can extract the keys/certificates of interest and save into a new file.

Errors Could not authenticate user 'null' on the ServiceNow instance The MID server runs tests when it first starts. One of the tests checks the connection from the MID server to the instance as well as permissions for the MID server user. This error is returned when the test fails. The user should be what is configured in the config.xml file. The error returns user 'null' since setting up mutual authentication on the MID server removes the mid.instance.username parameter. This error is usually a side effect of mutual authentication failure.

Quebec User Client Certificate UI action "Validate Stores/Certificates" error "Invalid client_cert: Unable to load certificate ,pem" The UI action checks field pem_certificate which is not populated for User Client Certificates and therefore the error. This UI Action is for the CA Certs only. Newer releases have removed this UI Action from User Client Certificates.

Failed in client cert chain verification/Certificate chain verification fail for incomplete certificate chain The sys_ca_certificate record has field "active" set to false The intermediate or root certificates have not been uploaded to sys_ca_certificate Adding User Client Certificate (sys_user_certificate) fails with error "Unable to validate the certificate from CA" Was the certificate chain added to the sys_ca_certificate? Yes: Continue No: Add the certificate chain to sys_ca_certificate Check if the certificate chain was published to the ADCv2 load balancer Add column "Publish Status" to sys_ca_certificate Is the "Publish Status" set to "exists"? Yes: Review the PEM file to confirm all the necessary certificates are present No: Wait till the status changes to "exists" Note: This will take a few minutes after adding the chain to sys_ca_certificate MID Server Wrapper logs shows "Certificates": or No X.509 certificate for client authentication MID Security Policy for *.service-now.com not OOB Note: When this is the root cause, the MID server will at first connect to the instance successfully. However, will no longer be able to authenticate once the MID Security Policy is retrieved from the instance.

The following MID Security Policy must be set with OOB values in order for the MID server mutual authentication to work.

https:// .service-now.com/nav_to.do?uri=mid_cert_check_policy.do?sys_id=f30ab21a5b311010eff60e281d81c78c Solution:

Stop the MID Server Revert the policy back to OOB or set following options to true Certificate Chain Check Hostname Check Revocation Check Update the MID Server config.xml following parameters to true Start the MID server Certificate chain incomplete As part of the mutual authentication process the MID server receives a list of certificates from the instance (the root and intermediate certificates uploaded to sys_ca_certificate). The MID server uses the client certificate if it was signed by one of the certificates retrieved from the instance. In the following example extracted from the MID Server wrapper logs we see the list of certificates returned on a lab environment:

| "CertificateRequest": { | "certificate types": [...] | "supported signature algorithms": [...] | "certificate authorities": [CN=CS Automation Lab Certificate Authority Intermediate, OU=CS Automation Lab Certificate Authority, O=CS Automation Lab, ST=Florida, C=US, CN=CS Automation Lab Certificate Authority Root, OU=CS Automation Lab Certificate Authority, O=CS Automation Lab, L=Orlando, ST=Florida, C=US, ... ] | } | ) In the above example we see both the Intermediate and Root certificates are received by the MID server. The authentication completes successfully in this example. However, the MID Server would return an empty client certificate if certificate chain received by the MID server was empty or incomplete.

Solution:

Navigate to sys_ca_certificate Ensure all necessary certificates have been uploaded and have Publish Status = "exists" Adding the client certificate to the MID server was not successful To resolve the issue:

Stop the MID server Backup .\agent\security\agent_keystore into another directory Restart the MID server to generate a new agent_keystore Stop the MID server Follow steps to add client certificate and private key to MID server Enable mutual authentication Start the MID server The certificate added to the MID server is not a client certificate You can see the certificate type by running command:

openssl x509 -in certificatePath.cert.pem -text -noout In the following example we see the Extended Key Usage for a client certificate:

X509v3 Extended Key Usage:

TLS Web Client Authentication, E-mail Protection While in the following example we see the Extended Key Usage for a server certificate:

X509v3 Extended Key Usage:

TLS Web Server Authentication Note: The certificate should be ok it the Extended Key Usage contains "TLS Web Client Authentication".

Solution:

Request a client certificate and private key from your certificate provider Stop the MID server Follow steps to add client certificate and private key to MID server Enable mutual authentication Start the MID server Q&A Are self signed certificates supported? No. Self signed certificates are not supported at this point.

Can you invalidate a MID server which is using mutual authentication? No.

Can you rekey a MID server using mutual authentication? No.

How do you know a MID server is using mutual authentication? Check the MID server field "Is using mutual authentication?"

Why is there no attachment under sys_user_certificate and why is field pem_certificate empty? The certificate is converted into a sha256 fingerprint and the attachment is deleted. This is expected behavior.

Will all my MID server need to use mutual authentication once I enable mutual authentication? No. You can still have mid servers using basic authentication.

How can I confirm my certificate chain is valid: Using OpenSSL verify command to confirm that the certificate chain is valid. Example command:

openssl verify -CAfile rootCert.pem \ -untrusted intermediateCert.pem \ clientCert.pem Alternatively, you can use the issuer hash to confirm the chain is valid. Example command:

openssl x509 -hash -issuer_hash -noout -in .pem In the above, we check the hash for the leaf certificate which was added to the MID server. This way we can check the chain.

In the following example we see the hashes for a test environment:

openssl x509 -hash -issuer_hash -noout -in UserExampleCert.pem 70f02020 20403030

openssl x509 -hash -issuer_hash -noout -in IntermediateCert.pem 20403030 20d05000

openssl x509 -hash -issuer_hash -noout -in RootCert.pem 20d05000 20d05000 Note that in the above we can see the chain: 70f02020 >>> 20403030 >>> 20d05000

Problems (PRB1556363) - Invalid and misleading error message: "MID Server encryption keys do not match and are no longer valid. To restore proper functionality, invalidate and re-validate the MID Server" for mTLS enabled MID Servers Additional Information How to configure inbound Certificate Based Authentication (mutual authentication) in ServiceNow