On some hosts MID Servers need more than 5 chances to start up the JVM within 30 seconds, leaving them Down during restarts/upgrades - Known Error

On some hosts MID Servers need more than 5 chances to start up the JVM within 30 seconds, leaving them Down during restarts/upgrades Description When a MID Server service starts, the Tanuki Wrapper process runs, which then has to start the Java JVM process (java.exe), which in turn runs the MID Server application. In the situation where apparently the "Wrapper Process has not received any CPU time", the Wrapper gives the JVM only 30 seconds to start. It will then only retry another 4 times (5 attempts in total), then give up, leaving the MID Server Down.

A manual start of the MID Server service is then needed, which requires access to the Host server, often requiring several attempts to start the service before the JVM starts within the 30 second timeout.

This is seen in 'soft' restarts for e.g. re-key, where only the JVM is restarted, and also 'hard' restarts from the 'restart MID' related link and where the service is stopped and started.

This problem fix should set the default wrapper property wrapper.max_failed_invocations to more than 5, as where the start time is variable and sometimes does happen with 30 seconds this should avoid a lot of the failures. Ideally also fix the root cause of the apparent CPU starvation, and extend the timeout for each attempt to more than 30 seconds.

Unconfirmed, but it is also possible that a Keystore issue might be indirectly caused by this problem, where a MID server restarts as part of a restart or upgrade, where it is then expecting to immediately create a new keypair on startup, and due to the delay caused by having to be manually started later, that process goes wrong and the keys don't match.

Steps to Reproduce

Exactly how to set up a windows server from scratch to reproduce this on demand is currently not known.

Restart a MID Server, either from the service, from the MID Server form, during Re-Key, validate/invalidate, or Upgrades. It ends up Down. This has been seen on Azure VMs, where the Windows OS is reporting plenty of free CPU at the time. Reproducible for one customer on Yokohama Patch 7 Hot Fix 2a, with wrapper 3.5.57 and JRE 17.0.12-sncmid1+3, on Windows Server 2022. Also Washington DC Patch 10 Hot Fix 2, and others. This might be due to resource issues.

Actual behaviour:

The wrapper can't start the JVM and MID Server application and leaves the MID Server down. That in turn causes an outage for any features using exclusively that MID Server, or performance degradations if failover/load balancing is implemented.

Setting wrapper.max_failed_invocations=60 in wrapper-override.conf does allow more retires, which can eventually work, however setting wrapper.startup.timeout or wrapper.ping.timeout does not extend the 30 second timeout in this situation.

Expected behaviour:

The MID Server was up, and is expected to still be up afterwards.

When it completely fails, this will be seen in the wrapper log:

2025/11/21 14:20:56 | There were 5 failed launches in a row, each lasting less than 300 seconds. Giving up. 2025/11/21 14:20:56 | There may be a configuration problem: please check the logs.

Example Wrapper log for it making several attempts before working. This has wrapper.debug=true in wrapper-override.conf, and edited for the interesting bits:

These are the 5s interval pings between JVM and Wrapper, while the MID Server was up and running normally, then the shutdown of the JVM, then the wrapper stopping. We stopped the wrapper service too. This was not a soft restart of just the JVM. 2025/12/05 14:46:10 | Send a packet PING : ping 00abbaf0 2025/12/05 14:46:10 | read a packet PING : ping 00abbaf0 2025/12/05 14:46:12 | Stopping the ServiceNow MID Server XXX service... 2025/12/05 14:46:15 | JVM signaled that it was stopped. 2025/12/05 14:46:16 | JVM exited normally. 2025/12/05 14:46:16 | <-- Wrapper Stopped

Then starting the wrapper again. This shows we had set wrapper.ping.timeout/wrapper.startup.timeout without it helping. 8 CPUs, 1GB heap.:

2025/12/05 14:46:17 | The "wrapper.ping.timeout" property was redefined on line #68 of configuration file: C:\ServiceNow\XXX\agent\conf\wrapper-override.conf 2025/12/05 14:46:17 | New Value wrapper.ping.timeout=300 2025/12/05 14:46:18 | Starting the ServiceNow MID Server XXX service... 2025/12/05 14:46:18 | --> Wrapper Started as Service 2025/12/05 14:46:18 | Java Service Wrapper Professional Edition 64-bit 3.5.57 2025/12/05 14:46:18 | Startup thread started. 2025/12/05 14:46:18 | PID: 10696 2025/12/05 14:46:18 | Wrapper configuration properties BEGIN: 2025/12/05 14:46:18 | F | | wrapper.java.maxmemory | 1024 2025/12/05 14:46:18 | F | | wrapper.ping.timeout | 300 2025/12/05 14:46:18 | F | | wrapper.startup.timeout | 600 2025/12/05 14:46:18 | P---- | NUMBER_OF_PROCESSORS=8

running java.exe just to get the version:

2025/12/05 14:46:18 | Java Command Line (Java Version): 2025/12/05 14:46:18 | Command: "jre\bin\java" -version 2025/12/05 14:46:19 | OpenJDK Runtime Environment (build 17.0.12-sncmid1+3)

Then running the JVM, which fails:

2025/12/05 14:46:19 | Java Command Line (Bootstrap): 2025/12/05 14:46:19 | Command: "jre\bin\java" -classpath "./lib/java-...." -Dfile.encoding=Cp1252 org.tanukisoftware.wrapper.bootstrap.WrapperBootstrap 1 com.service_now.mid.Main 1 2025/12/05 14:46:23 | Waiting to start... 2025/12/05 14:46:28 | Waiting to start... 2025/12/05 14:46:33 | Waiting to start... 2025/12/05 14:46:38 | Waiting to start... 2025/12/05 14:46:43 | Waiting to start... 2025/12/05 14:46:48 | Waiting to start... 2025/12/05 14:46:50 | Child process: Bootstrap: timed out 2025/12/05 14:46:50 | Timed out waiting for JVM process (Bootstrap). 2025/12/05 14:46:50 | Wrapper Process has not received any CPU time for 31 seconds. Extending timeouts. 2025/12/05 14:46:50 | Preparing to restart with mode 2. 2025/12/05 14:46:50 | JVM was running for 31 seconds (less than the successful invocation time of 300 seconds). 2025/12/05 14:46:50 | Incrementing failed invocation count (currently 1). 2025/12/05 14:46:50 | Enqueue Event 'jvm_failed_invocation' 2025/12/05 14:46:50 | Waiting 5 seconds before launching another JVM.

That was only given 14:46:19 to 14:46:50 = 31 seconds.

Where it said "Extending timeouts.", that doesn't seem to actually happen for the next try.

There was then a 2nd, 3rd, 4th, 5th attempt which also timed out at 30s. With a default configuration, that would have been it, leaving the MID Server down.

6th attempt managed it in 14 seconds, before the timeout, getting the MID server Up. So for this host carrying on retrying does eventually work. If the default reties had been higher, there would have been no outage, just a longer restart than expected.

2025/12/05 14:49:18 | Waiting to start... 2025/12/05 14:49:20 | Java Command Line (Java Version): 2025/12/05 14:49:20 | Command: "jre\bin\java" -version 2025/12/05 14:49:20 | openjdk version "17.0.12-sncmid1" 2024-07-16 2025/12/05 14:49:20 | OpenJDK Runtime Environment (build 17.0.12-sncmid1+3) 2025/12/05 14:49:20 | OpenJDK 64-Bit Server VM (build 17.0.12-sncmid1+3, mixed mode) 2025/12/05 14:49:20 | Java version: 17.0.12 2025/12/05 14:49:20 | Java vendor: OpenJDK 2025/12/05 14:49:20 | Java bits: 64-Bit 2025/12/05 14:49:20 | Java Command Line (Bootstrap): 2025/12/05 14:49:20 | Command: "jre\bin\java" -classpath "./lib/java-service-w..." -Dfile.encoding=Cp1252 org.tanukisoftware.wrapper.bootstrap.WrapperBootstrap 1 com.service_now.mid.Main 1 2025/12/05 14:49:23 | Waiting to start... 2025/12/05 14:49:28 | Waiting to start... 2025/12/05 14:49:33 | Waiting to start...

and it started. 14:49:20 - 14:49:34 = 14s However "Wrapper Process has not received any CPU time" logging is still seen after this point too, as it runs the MID server applciation.

2025/12/05 14:49:34 | Dec 05, 2025 2:49:34 PM com.glide.util.GlideProperties refreshProperties 2025/12/05 14:49:36 | Configuring log handler: com.service_now.mid.logging.MIDLogFileHandler 2025/12/05 14:49:36 | package: com.service_now.mid 2025/12/05 14:49:36 | Java Command Line (Dry Run): 2025/12/05 14:49:36 | Command: "jre\bin\java" --dry-run -Djava.util.logging.... org.tanukisoftware.wrapper.WrapperStartStopApp com.service_now.mid.Main 1 start com.service_now.mid.Main true 1 stop -- 2025/12/05 14:49:38 | Wrapper Process has not received any CPU time for 17 seconds. Extending timeouts. 2025/12/05 14:49:38 | Launching a JVM... 2025/12/05 14:49:38 | Java Command Line: 2025/12/05 14:49:38 | Command: "jre\bin\java" -Djava.util.logging.config.file=... org.tanukisoftware.wrapper.WrapperStartStopApp com.service_now.mid.Main 1 start com.service_now.mid.Main true 1 stop -- 2025/12/05 14:49:38 | Waiting to start... 2025/12/05 14:49:38 | JVM started (PID=6260) 2025/12/05 14:49:42 | WrapperJNI Debug: Java Executable: C:\ServiceNow\XXXagent\jre\bin\java.exe 2025/12/05 14:49:42 | WrapperJNI Debug: Windows version: 10.0.26100 2025/12/05 14:49:42 | WrapperManager Debug: Java PID : 6260 2025/12/05 14:49:42 | WrapperManager Debug: Java Version : 17.0.12-sncmid1+3 OpenJDK 64-Bit Server VM 2025/12/05 14:49:42 | WrapperManager Debug: OS Name : Windows Server 2022 2025/12/05 14:49:42 | WrapperManager Debug: Startup runner thread started. 2025/12/05 14:49:43 | Waiting to start... 2025/12/05 14:49:43 | Configuring log handler: com.service_now.mid.logging.MIDLogFileHandler 2025/12/05 14:49:43 | WrapperManager Debug: Open socket to Wrapper...Wrapper-Connection 2025/12/05 14:49:44 | accepted a socket on port 32001 from 127.0.0.1 at port 31003 2025/12/05 14:49:44 | closing backend server. 2025/12/05 14:49:44 | Start Application. 2025/12/05 14:49:44 | WrapperStartStopApp Debug: start(args) Will wait up to 2 seconds for the main method to complete. 2025/12/05 14:49:44 | WrapperStartStopApp Debug: invoking start main method 2025/12/05 14:49:44 | FIPS Enforced Mode: false 2025/12/05 14:49:44 | JVM property java.security.egd: file:/dev/urandom 2025/12/05 14:49:46 | JVM signaled that it was started. 2025/12/05 14:49:47 | ServiceNow MID Server XXX service started. 2025/12/05 14:50:00 | WARNING: A terminally deprecated method in java.lang.System has been called

Workaround This problem is currently under review and targeted to be fixed in a future release. Subscribe to this Known Error article to receive notifications when more information will be available.

Setting wrapper.max_failed_invocations=60 in wrapper-override.conf does allow more retires, which should eventually work, however setting wrapper.startup.timeout or wrapper.ping.timeout does not extend the 30 second timeout in this situation.