IBM Support

IBM Java for AIX HowTo: Track Java heap usage with HPROF

Technote (FAQ)


Question

IBM Java for AIX HowTo: Track Java heap usage with HPROF

Answer


The purpose of this document is to help understand and identify the source of object allocations in the Java heap and also to track down objects with excessive heap usage or leak for IBM Java for AIX.

This article shows how to generate HPROF format heapdump, interpret the output and the tools to examine and diagnose the state of your application.
Jump to section: ( Prepare) ( Configure) ( Collect) ( Interpret) Tools)
The instructions in this document make references to generic terms in Italics that will need to be replaced with information specific to the support call and the environment. It is very important that consistent and accurate values be used in place of the Italicized generic terms when collecting the data to ensure the prompt and correct delivery of the data when uploaded.
Generic Term Replace with
USERID The AIX userid running the Java process (e.g. wasadmin or root).
JAVA_PID The process id of the active Java process (e.g. use "ps" command to check the PID column to identify the process).
START_PATH The directory from which the Java process was started (e.g. /usr/IBM/WebSphere).
SPECIFIC_PATH User specificed Directory. (e.g., /tmp)
Overview
Step-by-Step Instructions

Step 1:

Prepare

The default PHD heapdump is smaller in size but lacks additional details for objects, thread, fields. For most of the scenarios, the information in PHD format heapdump is enough to diagnose and examine the state of the application. But, in some scenarios, due the missing object details, it is not possible to identify the source for the objects with leak or most memory usage.

The HPROF is a demonstration profiling tool shipped with the IBM® SDK that uses the JVMTI to collect and record information about Java™. It profilesJava heap and cpu. By tracking each object allocation, the HPROF format of heapdump helps in identifying the call stack for the object allocation and also the specific class/method allocating the object.

The HPROF format contains all the data present in the IBM PHD format plus additional details. As the HPROF heapdump tracks each object allocation and writes out the call stack allocating object, depending on the rate of object alocations there could be some impact to the application performance and memory usage.

The HPROF heapdumps are the size of the used Java heap. Make sure the directory where the heapdumps get written have enough disk space, so, the generated output file is not truncated.


A. Set the user ulimits:

From a command prompt and while logged in as the root user, execute following commands :

{ set file, data ulimit sizes to unlimited }
# chuser fsize=-1 data=-1 USERID


B. Perform the following actions in order for the changes to take effect:

- Stop the application
- Relogin as the "USERID" used in Step 1.A

- Confirm the new ulimits are in effect by executing the command:

# ulimit -a


- Do not restart the application until Step 2. Configure has been completed

Step 2:

Configure

A. Add java command line options:


To enable tracking of java heap object allocations, add:

-Xrunhprof:format=a,file=java.hprof.txt,heap=sites,thread=y,depth=8


To process the HPROF output file with the heap analysis tools, generate binary format output file by setting format to "b" for binary. The default is ASCII format.

With "heap=sites", the output file shows sorted list of sites with the most heavily allocated objects at the start.

The Depth option indicates number of method frames to display in a stack trace. The default is 4.

Use the file option to change the name of the output file. The default name for an ASCII file is "java.hprof.txt" and the default name for a binary file is "java.hprof".


Refer to URL below for additional details about the options:

https://www.ibm.com/support/knowledgecenter/SSYKE2_8.0.0/com.ibm.java.aix.80.doc/diag/tools/hprof.html#hprof


By default,

To designate a directory for the HPROF generated output file and and javacore, instead of the default directory, add:

-Xdump:directory=SPECIFIC_PATH



C. Restart the java application (e.g., node agent/manager) from the USERID new login session.

Step 3:

Collect

The HPROF profiling information doesn't get written out until the Java process exits either normally or with Out Of Memory exception. To get the output file while the Java process is still running, run command:

# kill -3 JAVA_PID



The HPROF output file, by default, gets generated in the START_PATH or in the SPECIFIC_PATH if set.


The directoryJava process was started from can be determined with command:

# procwdx JAVA_PID


For situations with continuous increase in Java heap, if the Java process is allowed to continue until Java heap is exhausted, the Java process ends with Out Of Memory exception, for examples as below:

JVMDUMP039I Processing dump event "systhrow", detail "java/lang/OutOfMemoryError" at 2017/03/18 21:48:15 - please wait.

.....

Dumping allocation sites ... done.



and by default following files get generated in the START_PATH or in the SPECIFIC_PATH if set :

Snap trace
AIX binary core
PHD format heapdump
javacore


Step 4:

interpret

The format and contents of the genrerated HPROF output file depends on the selected options.The header section at the beginning of the ASCII format of HPROF output file has explanation of the different sections in the output file.


Each trace record repesents the call stack which allocated the objects in the heap: Here is a sample trace record showing the application class and method that allocated the object in the heap are class named "OOM" and the method named "fillHeap":

TRACE 300745: (thread=200001)
java.lang.String. (String.java:332)
java.lang.String. (String.java:303)
java.lang.Long.toString(Long.java:411)
java.lang.Long.toString(Long.java:1044)
OOM.fillHeap(JavaOOM.java:16)


At the bottom of output file is a sorted list of allocations in descending order starting with the most heavily allocated object types and the trace number for the call stack where those allocations occurred.

For example :

SITES BEGIN (ordered by live bytes) Sat Mar 18 21:49:17 2017

percent live alloc'ed stack class
rank self accum bytes objs bytes objs trace name

1 46.14% 46.14% 221522424 9230101 221522424 9230101 300745 java.lang.String
2 46.12% 92.26% 221442424 9230101 221442424 9230101 300746 char[]
3 7.69% 99.95% 36920408 1 110508448 15 300753 java.lang.Object[]


SITES END


For detailed explanation of the output, refer to " Explanation of the HPROF output file " at :
https://www.ibm.com/support/knowledgecenter/en/SSYKE2_8.0.0/com.ibm.java.aix.80.doc/diag/tools/hprof.html


Step 6:

ACTION

Step 7:

ACTION

Step 8:

ACTION

Step 9:

ACTION

Step 10:

ACTION

Step 11:

ACTION

Step 12:

ACTION

Step 13:

ACTION

Step 14:

ACTION

Step 15:

ACTION

Step 16:

ACTION

Step 17:

ACTION

Step 18:

ACTION

Step 19:

ACTION

Step 20:

ACTION

Step 21:

ACTION

Document Type: Instruction
Content Type: Howto
Hardware: all Power
Operating System: all AIX Versions
IBM Java: all Java Versions
Author(s): Rama Tenjarla
Reviewer(s): John Carver

Document information

More support for: IBM Java

Software version: Version Independent

Operating system(s): AIX

Reference #: T1025034

Modified date: 21 March 2017