How to Resolve Java Crashes and Java Out of Memory Errors on Edge Encryption Proxies - Support and Troubleshooting

How to Resolve Java Crashes and Java Out of Memory Errors on Edge Encryption Proxies Description The following errors can occur related to the Java Virtual Machine (JVM) in either the edgeencryption.log or wrapper_ .log in the /logs directory of the Edge Proxy:

2017-06-15 20:58:33,091 ERROR Java heap space 2017-06-15 20:58:44,931 ERROR Java heap space 2017-06-15 21:00:11,359 ERROR Java heap space or

2016-11-29 09:18:26,765 [sn221104245-44485] WARN org.eclipse.jetty.util.thread.QueuedThreadPool - Unexpected thread death: org.eclipse.jetty.util.thread.QueuedThreadPool$3@3ad8d4eb in qtp221104245{STARTED,8 2016-11-29 09:18:28,672 [sn221104245-44387] WARN com.snc.edgeencryption.EdgeEncryptionLog - GetRequestTransformer exception java.lang.OutOfMemoryError: GC overhead limit exceeded 2016-11-29 09:18:34,875 [sn221104245-44342] WARN org.eclipse.jetty.servlet.ServletHandler - Error for /amb/connect java.lang.OutOfMemoryError: GC overhead limit exceeded 2016-11-29 09:18:35,813 [sn221104245-44344] WARN org.eclipse.jetty.servlet.ServletHandler - Error for /amb/handshake java.lang.OutOfMemoryError: Java heap space 2016-11-29 09:18:39,703 [sn221104245-44337] WARN org.eclipse.jetty.servlet.ServletHandler - Error for /angular.do java.lang.OutOfMemoryError: Java heap space 2016-11-29 09:18:55,719 [sn221104245-44338] WARN org.eclipse.jetty.servlet.ServletHandler - Error for /amb/connect java.lang.OutOfMemoryError: GC overhead limit exceeded 2016-11-29 09:18:31,094 [sn221104245-44532] ERROR com.snc.edgeencryption.EdgeEncryptionLog - GC overhead limit exceeded 2016-11-29 09:18:55,719 [sn221104245-44484] ERROR com.snc.edgeencryption.EdgeEncryptionLog - GC overhead limit exceeded or

INFO | jvm 64 | 2017/06/01 09:29:38.610 | [CodeBlob (0x0000000001af2290)] INFO | jvm 64 | 2017/06/01 09:29:38.610 | Framesize: 0 INFO | jvm 64 | 2017/06/01 09:29:38.610 | BufferBlob (0x0000000001af2290) used for StubRoutines (2) INFO | jvm 64 | 2017/06/01 09:29:40.033 | # INFO | jvm 64 | 2017/06/01 09:29:40.033 | # A fatal error has been detected by the Java Runtime Environment: INFO | jvm 64 | 2017/06/01 09:29:40.033 | # INFO | jvm 64 | 2017/06/01 09:29:40.033 | # Internal Error (sharedRuntime.cpp:834), pid=3856, tid=3824 INFO | jvm 64 | 2017/06/01 09:29:40.033 | # fatal error: exception happened outside interpreter, nmethods and vtable stubs at pc 0x0000000001af4a10 INFO | jvm 64 | 2017/06/01 09:29:40.033 | # INFO | jvm 64 | 2017/06/01 09:29:40.033 | # JRE version: Java(TM) SE Runtime Environment (8.0_40-b26) (build 1.8.0_40-b26) INFO | jvm 64 | 2017/06/01 09:29:40.033 | # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.40-b25 mixed mode windows-amd64 compressed oops) INFO | jvm 64 | 2017/06/01 09:29:40.033 | # Core dump written. Default location: C:\EdgeProxy\ServiceNow_443\bin\hs_err_pid3856.mdmp INFO | jvm 64 | 2017/06/01 09:29:40.033 | # INFO | jvm 64 | 2017/06/01 09:29:40.033 | # An error report file with more information is saved as: INFO | jvm 64 | 2017/06/01 09:29:40.033 | # C:\EdgeProxy\ServiceNow_443\bin\hs_err_pid3856.log INFO | jvm 64 | 2017/06/01 09:29:40.033 | # INFO | jvm 64 | 2017/06/01 09:29:40.033 | # If you would like to submit a bug report, please visit: INFO | jvm 64 | 2017/06/01 09:29:40.033 | # http://bugreport.java.com/bugreport/crash.jsp INFO | jvm 64 | 2017/06/01 09:29:40.033 | # ERROR | wrapper | 2017/06/01 09:29:40.142 | JVM exited unexpectedly. STATUS | wrapper | 2017/06/01 09:29:44.739 | Launching a JVM... INFO | jvm 65 | 2017/06/01 09:29:44.959 | Wrapper (Version 3.2.3.SNC.1) http://wrapper.organization.org INFO | jvm 65 | 2017/06/01 09:29:44.959 | Copyright 1999-2006 Company, Inc. All Rights Reserved. INFO | jvm 65 | 2017/06/01 09:29:44.959 | INFO | jvm 65 | 2017/06/01 09:29:44.959 | WARNING - The Wrapper jar file currently in use is version "3.2.3.SNC.1" INFO | jvm 65 | 2017/06/01 09:29:44.959 | while the version of the Wrapper which launched this JVM is INFO | jvm 65 | 2017/06/01 09:29:44.959 | "3.2.3". INFO | jvm 65 | 2017/06/01 09:29:44.959 | The Wrapper may appear to work correctly but some features may INFO | jvm 65 | 2017/06/01 09:29:44.959 | not function correctly. This configuration has not been tested INFO | jvm 65 | 2017/06/01 09:29:44.959 | and is not supported. INFO | jvm 65 | 2017/06/01 09:29:44.959 | INFO | jvm 65 | 2017/06/01 09:31:04.519 | >>> Total classes: 1,000 total length: 6,926,185 or

ERROR | wrapper | 2017/06/22 19:14:28.388 | JVM exited unexpectedly. STATUS | wrapper | 2017/06/22 19:14:28.396 | JVM exited in response to signal SIGKILL (9). STATUS | wrapper | 2017/06/22 19:14:32.540 | Launching a JVM...? INFO | jvm 4 | 2017/06/22 19:14:34.543 | Wrapper (Version 3.2.3.SNC.1) http://wrapper.organization.org INFO | jvm 4 | 2017/06/22 19:14:34.543 | Copyright 1999-2006 Company, Inc. All Rights Reserved. INFO | jvm 4 | 2017/06/22 19:14:34.543 | INFO | jvm 4 | 2017/06/22 19:14:35.952 | Jun 22, 2017 7:14:35 PM com.snc.da.cfg.GatewayScanner getTypesXmlResources INFO | jvm 4 | 2017/06/22 19:14:35.953 | INFO: Scanning for gateways using xml resource method INFO | jvm 4 | 2017/06/22 19:14:36.253 | Jun 22, 2017 7:14:36 PM com.snc.da.cfg.GatewayScanner getTypesMap Symptoms A blank browser screen appears when trying to reach the instance via the Edge Proxy, or an error message similar to "This site cannot be reached" is displayed on the browser screen Slow browser navigation when interacting with the instance via the Edge Proxy Release or Environment Edge Encryption Proxy

Cause The Edge Proxy Java Virtual Machine (JVM) may be running out of memory or crashing.

Resolution In order to resolve these JVM-related issues check and take the following actions.

Checking and upgrading, if necessary, the Java JRE that is used by the Edge Encryption proxy On the proxy, open the file: $Edge_Install_Directory/conf/wrapper.conf . Check the line containing " wrapper.java.command " it may look like the following:

wrapper.java.command=../java/jre/bin/java or wrapper.java.command=/home/oracle/jre1.8.0_121/bin/java or wrapper.java.command=C:\Program Files\Java\jre1.8.0_144\bin\java

Navigate to the bin directory as shown in the setting for wrapper.java.command. Find the version of the JRE by executing the command java -version

Linux:

./java -version java version "1.8.0_40" Java(TM) SE Runtime Environment (build 1.8.0_40-b27) Java HotSpot(TM) 64-Bit Server VM (build 25.40-b25, mixed mode)

Windows:

C:\istanbul_ee_install\istan_eep_443\java\jre\bin>java -version java version "1.8.0_40" Java(TM) SE Runtime Environment (build 1.8.0_40-b26) Java HotSpot(TM) 64-Bit Server VM (build 25.40-b25, mixed mode)

If steps 3-4 indicate a JRE version below 1.8.0_121, update the JRE to at least the 1.8.0_121 version by following the steps in the next section. Upgrading the JRE for Windows Navigate to the JRE download site at http://www.oracle.com/technetwork/java/javase/downloads/jre8-downloads-2133155.html . Accept the licensing agreement and download the appropriate version of the JRE for the operating system being used (for example, Windows x64 Offline -> jre-8u144-windows-x64.exe). After downloading, execute jre-8u144-windows-x64.exe, install using the GUI, and keep the older versions of the JRE if asked. Navigate to the installation directory, which is most likely: C:\Program Files\Java\jre1.8.0_xxx. If using 256-bit encryption keys, copy the following two jar files from the original java JRE installation directory into the new one.

From the old one at:

C:\ \java\jre\lib\security\local_policy.jar and C:\ \java\jre\lib\security\US_export_policy.jar

To the new one at:

C:\Program Files\Java\jre1.8.0_xxx\lib\security\local_policy.jar and C:\Program Files\Java\jre1.8.0_xxx\lib\security\US_export_policy.jar

Stop the Edge Proxy server. Modify the C:\ \conf\wrapper.conf file to point to the new location of the java installation:

wrapper.java.command=C:\Program Files\Java\jre1.8.0_xxx\bin\java

Save the wrapper.conf. The next time the proxy is started, it runs on the new JRE version.

Upgrading the JRE for Linux Navigate to the JRE download site at http://www.oracle.com/technetwork/java/javase/downloads/jre8-downloads-2133155.html . Accept the licensing agreement and download the appropriate version of the JRE for the operating system being used (for example, Linux x64 -> jre-8u144-linux-x64.tar.gz). Save the downloaded version to the Edge Proxy machines and open, unzip, and untar. The installation directory structure should look like this:

$directory_path/jre1.8.0_

If using 256-bit encryption keys, copy the following two jar files from the original java JRE installation directory into the new one.

From the old one at:

$Old_JRE_install_directory/java/jre/lib/security/local_policy.jar and $Old_JRE_install_directory/java/jre/lib/security/US_export_policy.jar

To the new one at:

$New_JRE_install_directory/java/jre/lib/security/local_policy.jar and $New_JRE_install_directory/java/jre/lib/security/US_export_policy.jar

Stop the Edge Proxy server. Modify the $Edge_Install_Directory/conf/wrapper.conf file to point to the new location of the java installation:

wrapper.java.command=$jre_install_directory/bin/java

(For example, wrapper.java.command=/home/sn/jre1.8.0_144/bin/java.) The next time the proxy is started, it runs on the new JRE version.

Adding the java startup property -XX:-UseAESIntrinsics Note: Do this step only if you did NOT follow the steps in the earlier section to upgrade the Java JRE that is used by the Edge Encryption proxy, since the startup property is not needed if the JRE is on 1.8.0_121 or later. Add wrapper.java.additional.4=-XX:-UseAESIntrinsics to the wrapper.conf files for each proxy. (Note that the "4" in the property can be changed to a higher unused number if you are already using "4" for a different java startup property.)

Instructions can be found in:

KB0622015: Edge Encryption proxy goes to 95-100% CPU and fails with Java exception .

After the wrapper.conf files are modified, the proxies must be restarted.

Ensuring that the proxies are scaled correctly and have enough JVM heap memory allocated The recommended requirements for the proxies are explained in the following product documentation pages:

Edge Encryption system requirements Sizing your Edge Encryption environment Set the proxy server initial and upper bound memory limits When setting the initial and upper bound memory limits, allocate 6GB (ideally) to the proxy JVMs or at least 4GB (by default, the proxy is run with 2GB in the JVM). To increase the JVM memory follow this documentation:

To accommodate the 6GB of heap memory allocation for the proxy JVM, the machines hosting each proxy must have at least 8GB of RAM available. Leave at least 2GB to the operating system and 6GB to the proxy JVM.

To make the proxies run at 6GB, edit the wrapper.conf file at $Edge_Install_Directory/conf :

Comment out the wrapper.java.initmemory and wrapper.java.maxmemory properties in the wrapper.conf file. Add the following properties instead:

For startup RAM: wrapper.java.additional.(the next sequentially unused wrapper.java.additional number)=-Xms6144m For maximum RAM: wrapper.java.additional.(the next sequentially unused wrapper.java.additional number + 1)=-Xmx6144m As an example:

wrapper.java.additional.5=-Xms6144m wrapper.java.additional.6=-Xmx6144m This change requires a proxy restart.

To size the number of proxies needed, determine the number of proxies you need. For more information, see Sizing your Edge Encryption environment .

You must have a good idea of how many users are accessing the instance via the proxies at any peak time during the day. Keep in mind the number of proxies needed for "X," which is the number of simultaneous users. Add an extra proxy server for every X simultaneous users.

Checking if the Edge Proxy JVM is killed by the Linux Operating System If the edge proxy runs on Linux, check if the wrapper_ .log shows this “ SIGKILL (9)” error:

ERROR | wrapper | 2017/06/22 19:14:28.388 | JVM exited unexpectedly. STATUS | wrapper | 2017/06/22 19:14:28.396 | JVM exited in response to signal SIGKILL (9). STATUS | wrapper | 2017/06/22 19:14:32.540 | Launching a JVM... INFO | jvm 4 | 2017/06/22 19:14:34.543 | Wrapper (Version 3.2.3.SNC.1) http://wrapper.tanukisoftware.org INFO | jvm 4 | 2017/06/22 19:14:34.543 | Copyright 1999-2006 Tanuki Software, Inc. All Rights Reserved. INFO | jvm 4 | 2017/06/22 19:14:34.543 | > INFO | jvm 4 | 2017/06/22 19:14:35.952 | Jun 22, 2017 7:14:35 PM com.snc.da.cfg.GatewayScanner getTypesXmlResources INFO | jvm 4 | 2017/06/22 19:14:35.953 | INFO: Scanning for gateways using xml resource method INFO | jvm 4 | 2017/06/22 19:14:36.253 | Jun 22, 2017 7:14:36 PM com.snc.da.cfg.GatewayScanner getTypesMap If this error is displayed, check the /var/log/messages file for more information. For example, you may see:

Jun 22 19:14:28 eo-0ee971a048 kernel: Out of memory: Kill process 8405 (java) score 939 or sacrifice child Jun 22 19:14:28 eo-0ee971a048 kernel: Killed process 8405 (java) total-vm:9275688kB, anon-rss:7259600kB, file-rss:0kB, shmem-rss:0kB Jun 22 19:14:28 eo-0ee971a048 wrapper[2121]: JVM exited unexpectedly.