Linux MID server upgrade fails, and MID Server remains Down, on upgrade to Madrid - Tanuki wrapper service shutdown as 'TERM Trapped' - Known Error

Linux MID server upgrade fails, and MID Server remains Down, on upgrade to Madrid - Tanuki wrapper service shutdown as 'TERM Trapped' Description When upgrading a MID server hosted on a CentOS Linux server, and other Linux distributions, from London to Madrid or Madrid to Madrid, the upgrade fails.

The upgrade files are downloaded, extracted to tmp, and the agent log will show the ServiceNow Platform Distribution Upgrade. However the logs for that upgrade service will show it stopped before finishing, leaving the MID Server in a Down state.

TERM trapped. Shutting down. Steps to Reproduce

Install a Madrid MID Server using an older zip file than the instance version, or Pin an existing MID Server to a different version in order to force an upgrade Install it on the Linux machine as daemon service /bin/sh mid.sh install Start MID server service mid start Let the MID Server to auto upgrade to match to instance (or Pinned) version Observe in agent0.log.0 that it finished downloading zip file, extract zip content to temp file and observe last message 11/05/18 17:47:56 (660) MIDServer MID Server stopping 11/05/18 17:47:56 (801) MIDServer MID Server stopped Observe the wrapper.log and last message 2018/11/05 17:49:55 | WrapperStartStopApp Debug: stopping. Waiting for 1 threads to complete. 2018/11/05 17:49:55 | Shutdown failed: Timed out waiting for signal from JVM. 2018/11/05 17:49:56 | JVM did not exit on request, termination requested. 2018/11/05 17:49:56 | Signal trapped. Details: 2018/11/05 17:49:56 | signal number=17 (SIGCHLD), source="unknown" 2018/11/05 17:49:56 | Received SIGCHLD, checking JVM process status. 2018/11/05 17:49:56 | JVM received a signal SIGKILL (9). 2018/11/05 17:49:56 | socket read no code (closed?). 2018/11/05 17:49:56 | Closing backend socket. 2018/11/05 17:49:56 | JVM process is gone. 2018/11/05 17:49:56 | JVM process exited with a code of 1, setting the Wrapper exit code to 1. 2018/11/05 17:49:56 | JVM exited after being requested to terminate. 2018/11/05 17:49:56 | Observe the glide-dist-upgrade.log INFO | jvm 1 | 2018/11/05 17:49:56.214 | Nov 05, 2018 5:49:56 PM com.snc.dist.mid_upgrade.UpgradeMain missingOrCanDelete INFO | jvm 1 | 2018/11/05 17:49:56.214 | INFO: /root/hm/linux_agent2/bin/wrapper-linux-x86-64 cannot be deleted: /root/hm/linux_agent2/bin/wrapper-linux-x86-64 (Text file busy) INFO | jvm 1 | 2018/11/05 17:49:56.214 | Retrying in 1000ms... STATUS | wrapper | 2018/11/05 17:49:56.657 | TERM trapped. Shutting down . INFO | jvm 1 | 2018/11/05 17:49:56.757 | Nov 05, 2018 5:49:56 PM com.snc.dist.mid_upgrade.UpgradeMain$1 stop INFO | jvm 1 | 2018/11/05 17:49:56.757 | INFO: WrapperListener.stop() INFO | jvm 1 | 2018/11/05 17:49:57.158 | Nov 05, 2018 5:49:57 PM com.snc.dist.mid_upgrade.UpgradeMain missingOrCanDelete INFO | jvm 1 | 2018/11/05 17:49:57.158 | INFO: /root/hm/linux_agent2/bin/wrapper-linux-x86-64 is no longer locked after 125 checks. INFO | jvm 1 | 2018/11/05 17:49:57.158 | Nov 05, 2018 5:49:57 PM com.snc.dist.mid_upgrade.UpgradeMarker markService INFO | jvm 1 | 2018/11/05 17:49:57.158 | INFO: Added service `snc-platform-dist-upgrade--tmp-1541468756837-0-upgrade-wrapper` to upgrade marker file. INFO | jvm 1 | 2018/11/05 17:49:57.358 | Nov 05, 2018 5:49:57 PM com.snc.dist.mid_upgrade.UpgradeMain wipeDirs INFO | jvm 1 | 2018/11/05 17:49:57.358 | INFO: ** Debug Stale files ** TargetDirectory Path = /root/hm/linux_agent2 INFO | jvm 1 | 2018/11/05 17:49:57.358 | Nov 05, 2018 5:49:57 PM com.snc.dist.mid_upgrade.UpgradeMain wipeDirs INFO | jvm 1 | 2018/11/05 17:49:57.359 | INFO: ** Debug Stale files ** Files in TargetDirectory = wrapper-tsims-addendum-1.3.txt extlib work bin scripts stop.sh installer.bat ServiceNow Open Source Disclosure 062818LMS0.pdf conf lib jre config.xml EULA.pdf .DS_Store properties installer.sh midinstaller start.sh wrapper-development-license-1.3.txt etc README.doc keystore stop.bat package start.bat logs INFO | jvm 1 | 2018/11/05 17:49:57.417 | Nov 05, 2018 5:49:57 PM com.snc.dist.mid_upgrade.UpgradeMain wipeDirs INFO | jvm 1 | 2018/11/05 17:49:57.417 | INFO: ** Debug Stale files ** Files from Matching pattern = ServiceNow Open Source Disclosure 062818LMS0.pdf EULA.pdf README.doc INFO | jvm 1 | 2018/11/05 17:49:57.417 | Nov 05, 2018 5:49:57 PM com.snc.dist.mid_upgrade.UpgradeMain migrateToTarget INFO | jvm 1 | 2018/11/05 17:49:57.417 | INFO: Copying files to MID server installation path. STATUS | wrapper | 2018/11/05 17:49:57.618 | The message "TERM trapped. Shutting down" indicate upgrade did not finished and wrapper service shutdown unexpectedly.

Workaround This problem has been fixed in New York, which will include the Tanuki wrapper version to 3.5.36 to resolve the issue.

There is no workaround to prevent this happening, however once the problem has been detected, you can get the upgrade to finish by manually running the upgrade-wrapper/bin/glide-dist-upgrade.sh script.

Look in the agent log to find the tmp folder. e.g. 05/18/19 10:32:07 (333) AutoUpgrade.3600 Added marker `/tmp/1558168258322-0` to upgrade marker file. Within that folder you will find the upgrade-wrapper/bin/glide-dist-upgrade.sh script Edit that script to turn on "wrapper.ignore_signals" # If uncommented, causes the Wrapper to be shutdown using an anchor file. # When launched with the 'start' command, it will also ignore all INT and # TERM signals. IGNORE_SIGNALS=true Run it as sudo sudo ./glide-dist-upgrade.sh start The following line will be noted on the "glide-dist-upgrade.log", but this doesn't prevent the upgrade completing because we ignore it. TERM trapped, but ignored. If that is not possible, then a manual MID Server upgrade would be required.

After upgrade we need to re-install the service in NY.