MID Server down when using Oracle Java JDK JRE version 11.0.10 or 11.0.11DescriptionIf the MID Server's JRE is upgraded to Oracle 11.0.10 or 11.0.11, it will go down with instance connection certificate-related errors. Quebec ships with 11.0.8. Rome with 11.0.9.1. However newer patches of Oracle or OpenJDK Java 11 are officially support, and so should also work (in theory).Steps to Reproduce Method 1 - Upgrade JRE on an existing MID Server Install a MID Server, validate, and confirm it is working.Download and install Java SDK 11.0.10 (or 11.0.11)jdk-11.0.10_windows-x64_bin.exe from https://www.oracle.com/java/technologies/javase/jdk11-archive-downloads.htmlEdit wrapper-override.conf to point to that JRE instead of the bundled one. Uncomment and update this line:wrapper.java.command=C:\Program Files\Java\jdk-11.0.10\bin\javaRestart the MID Server service. Agent log shows the following (cropped for the key errors): 06/04/21 13:41:32 (608) main Logger config: root=INFO06/04/21 13:41:32 (722) WrapperStartStopAppMain Running under Java version: 11.0.10, java PID: 8676, args: start06/04/21 13:41:33 (361) WrapperStartStopAppMain JVM default socket factory: class sun.security.ssl.SSLSocketFactoryImpl06/04/21 13:41:33 (361) WrapperStartStopAppMain Initializing MID Server06/04/21 13:41:33 (490) MIDServer Creating injector...06/04/21 13:41:34 (480) MIDServer Using configuration: C:\MID_SERVER\JRE11.0.10_empdpiper_QP3\agent\config.xml06/04/21 13:41:35 (444) MIDServer Loaded credentials provider: com.service_now.mid.keypairs.provider.standard.StandardKeyPairsProvider06/04/21 13:41:40 (946) MIDServer Setting basic authentication with user mid_user06/04/21 13:41:40 (950) MIDServer WARNING *** WARNING *** Error loading key store, attempting to use legacy password06/04/21 13:41:40 (952) MIDServer WARNING *** WARNING *** Error loading key store with legacy password06/04/21 13:41:40 (952) MIDServer WARNING *** WARNING *** stream does not represent a PKCS12 key storejava.io.IOException: stream does not represent a PKCS12 key store at org.bouncycastle.jcajce.provider.ProvPKCS12$PKCS12KeyStoreSpi.engineLoad(Unknown Source) at java.base/java.security.KeyStore.load(KeyStore.java:1479)...06/04/21 13:41:40 (954) MIDServer WARNING *** WARNING *** 06/04/21 13:41:40 (954) MIDServer SEVERE *** ERROR *** Could not read the keystore for certificatescom.snc.automation_common.integration.exceptions.AutomationIOException: Unexpected IOException loading KeyStore, caused by: stream does not represent a PKCS12 key store...06/04/21 13:41:40 (954) MIDServer SEVERE *** ERROR *** Could not locate certificate informationcom.snc.automation_common.integration.exceptions.AutomationIOException: Unexpected IOException loading KeyStore, caused by: stream does not represent a PKCS12 key store at com.service_now.mid.keystore.provider.KeyStoreProvider.loadKeyStore(KeyStoreProvider.java:208)...Caused by: java.io.IOException: stream does not represent a PKCS12 key store at org.bouncycastle.jcajce.provider.ProvPKCS12$PKCS12KeyStoreSpi.engineLoad(Unknown Source)...06/04/21 13:41:40 (995) MIDServer Logger config: root=INFO06/04/21 13:41:40 (997) MIDServer Refreshing LoggerFactory cache06/04/21 13:41:41 (057) MIDServer ThreadPool-Interactive started with corePoolSize: 10, maxPoolSize: 10, maximumQueueSize: 4006/04/21 13:41:41 (066) MIDServer ThreadPool-Expedited started with corePoolSize: 20, maxPoolSize: 20, maximumQueueSize: 40006/04/21 13:41:41 (073) MIDServer ThreadPool-Standard started with corePoolSize: 25, maxPoolSize: 25, maximumQueueSize: 50006/04/21 13:41:41 (084) MIDServer ExtensionContainer ThreadPool started with corePoolSize: 25, maximumPoolSize: 25, maximumQueueSize: 500...06/04/21 13:41:42 (381) MIDServer OCSPCheck adding BouncyCastle provider at -106/04/21 13:41:42 (381) MIDServer OCSPCheckedCertificateCache build with max capacity 3206/04/21 13:41:42 (381) MIDServer OCSPRevokedCertificateCache build with max capacity 1606/04/21 13:41:42 (381) MIDServer OCSPTimeoutErrorCache build with max capacity 1606/04/21 13:41:42 (384) MIDServer WARNING *** WARNING *** org.apache.commons.httpclient.HttpException: Session contains no certificates - Untrusted06/04/21 13:41:42 (384) MIDServer SEVERE *** ERROR *** getRecords failed (org.apache.commons.httpclient.HttpException: Session contains no certificates - Untrusted)06/04/21 13:41:42 (386) MIDServer WARNING *** WARNING *** MIDRemoteGlideRecord.query failed, retrying in 10 seconds...06/04/21 13:53:10 (934) StartupSequencer SEVERE *** ERROR *** SOAP Request: <SOAP-ENV:Envelope xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:SOAP-ENC="http://schemas.xmlsoap.org/soap/encoding/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:tns="http://www.service-now.com/GetMIDInfo" xmlns:m="http://www.service-now.com" xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/" SOAP-ENV:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/"><SOAP-ENV:Body><m:execute></m:execute></SOAP-ENV:Body></SOAP-ENV:Envelope>06/04/21 13:53:10 (934) StartupSequencer SEVERE *** ERROR *** SOAP Response: Status code=0, Response body=null06/04/21 13:53:10 (934) StartupSequencer SEVERE *** ERROR *** Problem invoking InstanceInfo on https://empdpiper.service-now.com/: Please check that the InstanceInfo page exists in the sys_public table and active="true".06/04/21 13:53:10 (934) StartupSequencer SEVERE *** ERROR *** Request not sent to uri= https://empdpiper.service-now.com/InstanceInfo.do?SOAP : org.apache.commons.httpclient.HttpException: Session contains no certificates - Untrusted(Network Configuration issue) Please check that the MID server can ping the instance: https://empdpiper.service-now.com/You may also need to configure the network that the MID server uses to allow traffic over TCP port 443.06/04/21 13:53:10 (935) StartupSequencer SEVERE *** ERROR *** test failurejava.lang.IllegalStateException: Unable to connect to instance. at com.service_now.mid.services.StartupSequencer.runTests(StartupSequencer.java:630) at com.service_now.mid.services.StartupSequencer.startupSequencerRunnable(StartupSequencer.java:686) at java.base/java.lang.Thread.run(Thread.java:834) Method 2 - Start the MID Server for the first time, using the JRE from the beginning: Download and install Java SDK 11.0.10 (or 11.0.11)jdk-11.0.10_windows-x64_bin.exe from https://www.oracle.com/java/technologies/javase/jdk11-archive-downloads.htmlExtract a ZIP file installer, and edit wrapper-override.conf to point to that JRE instead of the bundled one. wrapper.java.command=C:\Program Files\Java\jdk-11.0.10\bin\javaEdit config.xml and wrapper-override.conf for the usual mandatory parameters/names.Run start.bat to run the MID Server for the first time If the wrapper-override.conf is pointed to the new JRE, before running start.bat for the first time, then the error is slightly different. That suggests the errors to do with the keystore are to do with upgrading the JRE after the MID server has already set up the keystore. However it still doesn't work. Agent log shows the following: 06/04/21 19:06:58 (180) main Logger config: root=INFO 06/04/21 19:06:58 (330) WrapperStartStopAppMain Running under Java version: 11.0.10, java PID: 13716, args: start06/04/21 19:06:58 (930) WrapperStartStopAppMain JVM default socket factory: class sun.security.ssl.SSLSocketFactoryImpl06/04/21 19:06:58 (930) WrapperStartStopAppMain Initializing MID Server06/04/21 19:06:59 (038) MIDServer Creating injector...06/04/21 19:07:00 (583) MIDServer Using configuration: C:\MID_SERVER\JRE11.0.10_empdpiper_QP3_from_install\agent\config.xml06/04/21 19:07:01 (580) MIDServer Loaded credentials provider: com.service_now.mid.keypairs.provider.standard.StandardKeyPairsProvider06/04/21 19:07:07 (147) MIDServer Setting basic authentication with user mid_user06/04/21 19:07:07 (150) MIDServer Keystore file keystore\agent_keystore.jks not found. Creating new keystore file.06/04/21 19:07:07 (208) MIDServer Logger config: root=INFO06/04/21 19:07:07 (210) MIDServer Refreshing LoggerFactory cache06/04/21 19:07:07 (299) MIDServer ThreadPool-Interactive started with corePoolSize: 10, maxPoolSize: 10, maximumQueueSize: 4006/04/21 19:07:07 (308) MIDServer ThreadPool-Expedited started with corePoolSize: 20, maxPoolSize: 20, maximumQueueSize: 40006/04/21 19:07:07 (319) MIDServer ThreadPool-Standard started with corePoolSize: 25, maxPoolSize: 25, maximumQueueSize: 50006/04/21 19:07:07 (344) MIDServer ExtensionContainer ThreadPool started with corePoolSize: 25, maximumPoolSize: 25, maximumQueueSize: 50006/04/21 19:07:08 (016) MIDServer MIDCredentialsConfigProvider initialized with com.service_now.mid.creds.provider.standard.StandardCredentialsProvider06/04/21 19:07:08 (685) MIDServer OCSPCheck adding BouncyCastle provider at -106/04/21 19:07:08 (685) MIDServer OCSPCheckedCertificateCache build with max capacity 3206/04/21 19:07:08 (685) MIDServer OCSPRevokedCertificateCache build with max capacity 1606/04/21 19:07:08 (686) MIDServer OCSPTimeoutErrorCache build with max capacity 1606/04/21 19:07:08 (689) MIDServer WARNING *** WARNING *** org.apache.commons.httpclient.HttpException: Session contains no certificates - Untrusted06/04/21 19:07:08 (689) MIDServer SEVERE *** ERROR *** getRecords failed (org.apache.commons.httpclient.HttpException: Session contains no certificates - Untrusted)06/04/21 19:07:08 (693) MIDServer WARNING *** WARNING *** MIDRemoteGlideRecord.query failed, retrying in 10 seconds06/04/21 19:07:18 (823) MIDServer WARNING *** WARNING *** org.apache.commons.httpclient.HttpException: Session contains no certificates - Untrusted06/04/21 19:07:18 (823) MIDServer SEVERE *** ERROR *** getRecords failed (org.apache.commons.httpclient.HttpException: Session contains no certificates - Untrusted)06/04/21 19:07:18 (824) MIDServer WARNING *** WARNING *** MIDRemoteGlideRecord.query failed, retrying in 15 seconds06/04/21 19:07:33 (958) MIDServer WARNING *** WARNING *** org.apache.commons.httpclient.HttpException: Session contains no certificates - Untrusted06/04/21 19:07:33 (958) MIDServer SEVERE *** ERROR *** getRecords failed (org.apache.commons.httpclient.HttpException: Session contains no certificates - Untrusted)06/04/21 19:07:33 (959) MIDServer WARNING *** WARNING *** MIDRemoteGlideRecord.query failed, retrying in 22 seconds...WorkaroundThis problem is currently under review. You can contact ServiceNow Technical Support or subscribe to this Known Error article by clicking the Subscribe button at the top right of this article to be notified when more information becomes available. Tests suggest this isn't a problem with pre-release Rome version builds. A code change made the java bug irrelevant. As a workaround, use JRE to 11.0.9 if possible, or apply the following customisation to each MID Server install if Java 11.0.10 or 11.0.11 is used: Fresh Install of a MID Server: Install JRE/JDK 11.0.10 or .11 on the MID server host, and make a note of the location. e.g. C:\Program Files\Java\jdk-11.0.11In the Java install folder: Open <jre path>\conf\security\java.security in a text editor. Check that this line is present, and 'true', and add it if necessary. Save, but keep it open. security.overridePropertiesFile=true Create a file "override-java.security" in the same <jre path>\conf\security\ folder.Paste into it all the listed providers, copied from the java.security file: ## List of providers and their preference orders (see above):#security.provider.1=SUNsecurity.provider.2=SunRsaSignsecurity.provider.3=SunECsecurity.provider.4=SunJSSEsecurity.provider.5=SunJCEsecurity.provider.6=SunJGSSsecurity.provider.7=SunSASLsecurity.provider.8=XMLDSigsecurity.provider.9=SunPCSCsecurity.provider.10=JdkLDAPsecurity.provider.11=JdkSASLsecurity.provider.12=SunMSCAPIsecurity.provider.13=SunPKCS11 Add this additional line to the end, using whatever number is next. e.g.: security.provider.14=org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider Download the ZIP file installer from the instance, and extract the "agent" folder to the MID Server install location.In the MID Server install folder: Open agent\config.xml in a text editor populate the 4 parameters in the "REQUIRED Parameters" section.If you use a proxy between MID Server and Instance, then uncomment and fill in that set of parameters too.Save. Open agent\conf\wrapper-override.conf in an editor. Fill in the service and wrapper name, appending the same "name" you chose in the config.xml file. e.g. if the config name parameter is "Prod Disco MID", the Windows Service section will look like: ################################################################################# Windows Service################################################################################# The following properties must be unique per MID installed on the same system.## REQUIRED: Name token of the servicewrapper.name=snc_mid_Prod Disco MID# REQUIRED: Display name of the servicewrapper.displayname=ServiceNow MID Server_Prod Disco MID Uncomment, and set the path for the external JRE e.g. ################################################################################# External JRE################################################################################# Uncomment and edit if an external JRE is preferred. By default,# the internal JRE distribution is used.## OPTIONAL: The path (relative to agent dir or absolute) to the java binwrapper.java.command=C:\Program Files\Java\jdk-11.0.11\bin\java Add this extra line at the end, pointing to the file you created within the java install folder earlier. wrapper.java.additional.<next_number_available>=-Djava.security.properties=<absolute path to override-java.security> for example (note: in this example we need quotation marks due to the spaced in the folder names): wrapper.java.additional.7=-Djava.security.properties="C:\Program Files\Java\jdk-11.0.11\conf\security\override-java.security" Save. Open a Command Prompt, as Administrator, Change directory to the the MID Server installation's agent folderType "start.bat", and enter. This will create the windows service (running as SYSTEM), and start the MID Server applicationWithin a minute, the MID Server record should appear in the instance (check agent log if it doesn't)Validate the MID Server You may want to later stop the service, change the Login as user to a non-admin account, and start it again. Notes: When modifying existing MID Server installations, some of the above steps can be skipped.If you patch Java again, and the Java install folder changes, a new override-java.security file will need creating within the new java install.Related Problem: PRB1502371