Edge Encryption Proxy Java Crash: V [libjvm.so+0x43b6d7] ciObjectFactory::create_new_metadata(Metadata*)+0x327 - Support and Troubleshooting

Edge Encryption Proxy Java Crash: V [libjvm.so+0x43b6d7] ciObjectFactory::create_new_metadata(Metadata*)+0x327 Issue  Symptoms An Edge Encryption proxy goes to an Unresponsive state unexpectedly and then shortly comes back to a state of Online.

Checking the logfile /logs/wrapper_ .log the following Java crash is seen at the time of the proxy going to Unresponsive:

INFO | jvm 1 | 2019/06/21 16:08:25.620 | # INFO | jvm 1 | 2019/06/21 16:08:25.620 | # A fatal error has been detected by the Java Runtime Environment: INFO | jvm 1 | 2019/06/21 16:08:25.620 | # INFO | jvm 1 | 2019/06/21 16:08:25.620 | # SIGSEGV (0xb) at pc=0x00007ff13021e6d7, pid=12334, tid=0x00007ff100ff7700 INFO | jvm 1 | 2019/06/21 16:08:25.620 | # INFO | jvm 1 | 2019/06/21 16:08:25.620 | # JRE version: Java(TM) SE Runtime Environment (8.0_144-b01) (build 1.8.0_144-b01) INFO | jvm 1 | 2019/06/21 16:08:25.620 | # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.144-b01 mixed mode linux-amd64 compressed oops) INFO | jvm 1 | 2019/06/21 16:08:25.620 | # Problematic frame: INFO | jvm 1 | 2019/06/21 16:08:25.620 | # V [libjvm.so+0x43b6d7] ciObjectFactory::create_new_metadata(Metadata*)+0x327 INFO | jvm 1 | 2019/06/21 16:08:25.620 | # INFO | jvm 1 | 2019/06/21 16:08:25.620 | # Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again INFO | jvm 1 | 2019/06/21 16:08:25.620 | # INFO | jvm 1 | 2019/06/21 16:08:25.620 | # An error report file with more information is saved as: INFO | jvm 1 | 2019/06/21 16:08:25.620 | # /sn/dir/application/servicenow/edge/bin/hs_err_pid12334.log INFO | jvm 1 | 2019/06/21 16:08:25.620 | # INFO | jvm 1 | 2019/06/21 16:08:25.620 | # Compiler replay data is saved as: INFO | jvm 1 | 2019/06/21 16:08:25.620 | # /sn/dir/application/servicenow/edge/bin/replay_pid12334.log INFO | jvm 1 | 2019/06/21 16:08:25.620 | # INFO | jvm 1 | 2019/06/21 16:08:25.620 | # If you would like to submit a bug report, please visit: INFO | jvm 1 | 2019/06/21 16:08:25.620 | # http://bugreport.java.com/bugreport/crash.jsp INFO | jvm 1 | 2019/06/21 16:08:25.620 | # ERROR | wrapper | 2019/06/21 16:08:39.779 | JVM exited unexpectedly. STATUS | wrapper | 2019/06/21 16:08:39.779 | JVM exited in response to signal UNKNOWN (6). STATUS | wrapper | 2019/06/21 16:08:43.885 | Launching a JVM... INFO | jvm 2 | 2019/06/21 16:08:54.005 | Wrapper (Version 3.2.3.SNC.1) http://wrapper.tanukisoftware.org INFO | jvm 2 | 2019/06/21 16:08:54.005 | Copyright 1999-2006 Tanuki Software, Inc. All Rights Reserved.

The key part of the logging is here:

V [libjvm.so+0x43b6d7] ciObjectFactory::create_new_metadata(Metadata*)+0x327

Release Applies to any release.

Cause The cause is discussed here:

https://confluence.atlassian.com/kb/application-crashes-with-sigsegv-jvm-dll-ciobjectfactory-create_new_metadata-808748283.html

Where there is a javaagent configured in the Edge Encryption proxy's Java Virtual Machine (JVM), most likely put there to monitor the JVM's performance.

To confirm that this is the case go to the /conf/wrapper.conf file and check for a JVM startup property that contains "javaagent" for example:

wrapper.java.additional.7 = -javaagent:/sn/dir/application/servicenow//myApp/myApp.jar

Note there may be additional JVM startup properties (wrapper.java.additional.x) related to the -javaagent:... defined above.

From the link:

"Cause The cause is not yet identified, there are a number of Java bugs that may be related to the issue (listed below). What we have identified so far that it does not only affect Atlassian applications, but Java application running Java 8 update 40 or later in general, in combination with Java agent(s) using Spring AOP instrumentation."

Resolution The addition of the javaagent is completely outside of the function of the Edge Encryption proxy. The interaction that caused the Java crash was between the javaagent and Java itself, not any interaction with the proxy itself.

Again referring to the link above for resolution:

https://confluence.atlassian.com/kb/application-crashes-with-sigsegv-jvm-dll-ciobjectfactory-create_new_metadata-808748283.html

"Workaround Please note that the workaround you apply is done at your own risk, you need to evaluate the possible impacts of either.

* Disable/remove Java agent from JVM startup arguments * See the agent-specific possible workarounds above * Contact your Java agent vendor"

Of course the easiest thing to do is to disable the javaagent if you are not actively using it for monitoring, i.e. commenting out all of the wrapper.java.additional.x lines related to the javaagent from the wrapper.conf file and restarting the proxies.

Without making any changes will maintain a risk of more of these JVM crashes.