IBM Java for AIX HowTo: Troubleshoot OOM due to Java Object Heap Exhaustion

There are different types of memory problems in Java that can throw an Out Of Memory(OOM) error. These Out Of Memory errors can be broadly classified into Java object heap exhaustion issues or Native Memory exhaustion issues.

This document mainly addresses troubleshooting the Out Of Memory errors caused by Java object heap exhaustion. Garbage-collection output (verbose:gc), javacore files, and heap dumps can be helpful in diagnosing these problems.

IBM's tooling and documentation can assist in the understanding, monitoring, and problem diagnosis of applications and deployments running IBM Runtime Environments. Using these tools to analyze the diagnostic data generated by IBM Java's superior diagnostic capabilities, enables the application developers to monitor and troubleshoot a wide range of issues.

IBM provides feature rich and free diagnostic tools to analyze the generated diagnostic data:

IBM Health Center Tool
IBM Garbage Collection and Memory Visualizer (GCMV)
IBM HeapAnalyzer Tool
IBM Memory Analyzer Tool

Support for these tools is only available from the forum and outside the scope of our support organization.

To confirm the OutOfMemory error is caused by Java object heap exhaustion, take the following steps:

1) Locate the following diagnostic data, generated automatically at the time of the OOM because of the default dump agents configured in IBM Java:

a. javacore.*.txt files (Java thread dump files)
b. Snap.*.trc files (Snap trace files)
c. heap.*dmp files (Heap dump files)
d. gc log

Each one of these files can be multiple in number. The default location or the changed location of these files can be understood by looking into this technote. If the diagnostic data has not been generated for some reason, skip to Generation of Diagnostic Data.

2) Check the SIGINFO line in the generated javacore files to confirm the java heap "Out Of Memory" error by executing the following command from a command prompt:

# egrep -h "1TISIGINFO|1TIDATETIME" javacore*

Output should be similar to:
1TISIGINFO Dump Event "systhrow" (00040000) Detail "java/lang/OutOfMemoryError" received
1TIDATETIME Date: 2016/09/20 at 17:03:44

3) Check the maximum size the heap can grow up to(Xmx value) from the javacore files by executing the following command from a command prompt:

# egrep -h "ARG.*Xm[xs]" javacore*

Output should be similar to:
2CIUSERARG -Xms3072m
2CIUSERARG -Xmx3072m
...............

To check the heap space that is free from the javacore.*.txt files, execute the following command from a command prompt:

# egrep -h "1STHEAPF" $( grep -l "1TISIGINFO.*java/lang/OutOfMemoryError" javacore*txt )

Output should be similar to:
1STHEAPFREE Bytes of Heap Space Free: 45530
1STHEAPFREE Bytes of Heap Space Free: 45530

(Convert the hexadecimal value of free heap bytes to decimal value)
Calculate the percentage of the of free heap against the maximum heap size and if that value is <4%, this is a confirmation that the OOM is due to Java object heap exhaustion. Proceed to the next sections for analyzing the data and troubleshooting some common causes of OOM due to Java object heap exhaustion.

If an OOM exception occurs when the Java object free space is over 20%, this implies the Out of Memory is due to native heap exhaustion. Do not continue with the next steps in this technote and refer to the following web pages to trouble shoot some common causes of OOM due to native heap exhaustion:

Troubleshooting native memory issues
Troubleshooting native memory issues when using 64-bit JVM

Some common causes for Java object heap Out of Memory (OOM) errors are :

Scenario 1: Unexpected large object allocation
Scenario 2: Java object heap configured too small
Scenario 3: Application memory leak

Please follow the information for each of these scenarios to identify the root cause of the Java object heap OOM.

Large object allocation requests from the application can cause OOM due to :

A. the large object size being requested is more than the available free Java object heap space.
B. the Java object heap does not have enough contiguous free heap to allocate the large object.

The JVM memory management component traces the size and class block address, which can be found in the Snap dump and Javacore files.

Check the Javacore for the allocation failure that caused the OOM by executing this command from a command prompt:

# grep "returning NULL" javacore* | cut -d')' -f2-

Output should be similar to:
returning NULL! 551550984 bytes requested for object of class
0000000040016800 from memory space 'Generational' id=0000010011035770

The type of object being allocated can be determined from the classes section of the Javadump by executing the following command from a command prompt:

# grep "3CLTEXTCLASS.*CLASS_BLOCK_ADDRESS" javacore* | cut -d: -f2- | sort -u

example:
# grep "3CLTEXTCLASS.*0000000040016800" javacore* | cut -d: -f2- | sort -u

Output should be similar to :
3CLTEXTCLASS [B(0x0000000040016800)

In the above example, the application is allocating a very large object (551 MB) that is associated with a byte array (see the symbols [B).

To resolve this large object allocation issue, the application's developer or owner will need to debug the object associated with this allocation.

The Java object heap becomes exhausted when the application does not release references to objects or delete the objects and hence GC cannot release the memory associated with these objects to make a new object allocation. Garbage collection can free only objects that are no longer referenced by other objects, or are referenced from the application thread stacks.

To understand the Java object heap utilization and GC and to troubleshhot the issue:

A) Download, install and run IBM Garbage Collection and Memory Visualizer(GCMV) tool.
B) Java heap exhaustion can be identified from the GCMV tool output when garbage collection occurs more frequently with less Java object heap being released. Eventually the JVM will fail, and the heap occupancy will be at, or almost at, 100%.
C) If the Java heap is being exhausted, increase the Java heap size by taking into consideration the current heap size, the work load and the application team's recommendations and see if that provides enough buffer to stop the OOMs from occurring. The Java object heap size can be increased by setting the -Xmx command line option to the desired value and restarting the application.

If , after increasing the Java heap size, the application continues to experience OOM exceptions, the owner or developer of the application code needs to examine the objects for any memory leaks. Java object memory leaks occur when the application does not release objects.

IBM JVM generates Heap dumps, which are dumps of all the live objects that are on the Java Object Heap, which are being used by the running Java application. To analyze the heapdumps that are in phd format and troubleshoot the issue:

A) Download and install the IBM Memory Analyzer tool. IBM Heap Analyzer can be used alternatively/along with IBM Memory Analyzer tool to analyze the heap dumps.
B) Analyze the heapdump files using the IBM Memory Analyzer Tool to obtain the summary of leak supects filling up the Java Object Heap.
C) The objects retained in the heap are dictated by the application logic and work load. Contact your application vendor to debug the application logic or obtain their recommendations based on the summary of leak suspects, the application needs and work load.

If the OOM due to Java Object Heap Exhaustion is not resolved at this point, please contact IBM.

Follow the steps below to generate the desired diagnostic data at the next occurrence of the OOM:

A) Add the following dump options to the Java command line which will disable generation of system core file(not needed to debug this issue) and enable the generation of the diagnostic data necessary to trouble shoot the OutOfMemory error:

-Xdump:system:none
-Xdump:java+snap+heap:events=systhrow,filter=java/lang/OutOfMemoryError,range=1..3

B) To specify a directory that has enough disk space to write the generated debug data instead of the default directory of START_PATH, add the following command line option:

-Xdump:directory=TMP_PATH

C) If not enabled already, generate verbose GC logging usually written to stderr and redirect standard error for the verbose GC data to a specific log file by adding the following command line options:

-verbose:gc
-Xverbosegclog:TMP_PATH/gc.log

D) Restart the Java application (e.g., node agent/manager) and recreate the OOM. The diagnostic data will be generated at the next occurence of the OOM.

If the generation of the diagnostic data is still , refer to this technote that provides the instrcutions on how to troubleshoot truncated, missing, or corrupted Diagnostic Files (e.g., javacore, heapdump, process core files). Once the diagnostic files have been generated go back to Common causes of OOM for troubleshooting common Java Object Heap OOM issues.

If, after following the above instructions, the application continues to generate OOM exceptions, please complete the following steps:

1. Confirm that you have completed all of the above steps.
2. Contact IBM and open a new IBM service request (new PMR).
3. Collect and upload data as per the data collection procedure for Java heap issues.