Question & Answer
Question
IBM Java for AIX HowTo: Track Java heap usage with HPROF
Answer
The purpose of this document is to help understand and identify the source of object allocations in the Java heap and also to track down objects with excessive heap usage or leak for IBM Java for AIX.
This article shows how to generate HPROF format heapdump, interpret the output and the tools to examine and diagnose the state of your application.
| Generic Term | Replace with |
| USERID | The AIX userid running the Java process (e.g. wasadmin or root). |
| JAVA_PID | The process id of the active Java process (e.g. use "ps" command to check the PID column to identify the process). |
| START_PATH | The directory from which the Java process was started (e.g. /usr/IBM/WebSphere). |
| SPECIFIC_PATH | User specificed Directory. (e.g., /tmp) |
Prepare
The default PHD heapdump is smaller in size but lacks additional details for objects, thread, fields. For most of the scenarios, the information in PHD format heapdump is enough to diagnose and examine the state of the application. But, in some scenarios, due the missing object details, it is not possible to identify the source for the objects with leak or most memory usage.
The HPROF is a demonstration profiling tool shipped with the IBM® SDK that uses the JVMTI to collect and record information about Java™. It profilesJava heap and cpu. By tracking each object allocation, the HPROF format of heapdump helps in identifying the call stack for the object allocation and also the specific class/method allocating the object.
The HPROF format contains all the data present in the IBM PHD format plus additional details. As the HPROF heapdump tracks each object allocation and writes out the call stack allocating object, depending on the rate of object alocations there could be some impact to the application performance and memory usage.
The HPROF heapdumps are the size of the used Java heap. Make sure the directory where the heapdumps get written have enough disk space, so, the generated output file is not truncated.
A. Set the user ulimits:
From a command prompt and while logged in as the root user, execute following commands :
{ set file, data ulimit sizes to unlimited }
# chuser fsize=-1 data=-1 USERID
B. Perform the following actions in order for the changes to take effect:
- Stop the application
- Relogin as the "USERID" used in Step 1.A
- Confirm the new ulimits are in effect by executing the command:
# ulimit -a
- Do not restart the application until Step 2. Configure has been completed
Configure
A. Add java command line options:
To enable tracking of java heap object allocations, add:
-Xrunhprof:format=a,file=java.hprof.txt,heap=sites,thread=y,depth=8
To process the HPROF output file with the heap analysis tools, generate binary format output file by setting format to "b" for binary. The default is ASCII format.
With "heap=sites", the output file shows sorted list of sites with the most heavily allocated objects at the start.
The Depth option indicates number of method frames to display in a stack trace. The default is 4.
Use the file option to change the name of the output file. The default name for an ASCII file is "java.hprof.txt" and the default name for a binary file is "java.hprof".
Refer to URL below for additional details about the options:
https://www.ibm.com/support/knowledgecenter/SSYKE2_8.0.0/com.ibm.java.aix.80.doc/diag/tools/hprof.html#hprof
By default,
To designate a directory for the HPROF generated output file and and javacore, instead of the default directory, add:
-Xdump:directory=SPECIFIC_PATH
C. Restart the java application (e.g., node agent/manager) from the USERID new login session.
Collect
The HPROF profiling information doesn't get written out until the Java process exits either normally or with Out Of Memory exception. To get the output file while the Java process is still running, run command:
# kill -3 JAVA_PID
The HPROF output file, by default, gets generated in the START_PATH or in the SPECIFIC_PATH if set.
The directoryJava process was started from can be determined with command:
# procwdx JAVA_PID
For situations with continuous increase in Java heap, if the Java process is allowed to continue until Java heap is exhausted, the Java process ends with Out Of Memory exception, for examples as below:
JVMDUMP039I Processing dump event "systhrow", detail "java/lang/OutOfMemoryError" at 2017/03/18 21:48:15 - please wait.
.....
Dumping allocation sites ... done.
and by default following files get generated in the START_PATH or in the SPECIFIC_PATH if set :
Snap trace
AIX binary core
PHD format heapdump
javacore
interpret
The format and contents of the genrerated HPROF output file depends on the selected options.The header section at the beginning of the ASCII format of HPROF output file has explanation of the different sections in the output file.
Each trace record repesents the call stack which allocated the objects in the heap: Here is a sample trace record showing the application class and method that allocated the object in the heap are class named "OOM" and the method named "fillHeap":
TRACE 300745: (thread=200001)
java.lang.String.
java.lang.String.
java.lang.Long.toString(Long.java:411)
java.lang.Long.toString(Long.java:1044)
OOM.fillHeap(JavaOOM.java:16)
At the bottom of output file is a sorted list of allocations in descending order starting with the most heavily allocated object types and the trace number for the call stack where those allocations occurred.
For example :
SITES BEGIN (ordered by live bytes) Sat Mar 18 21:49:17 2017
percent live alloc'ed stack class
rank self accum bytes objs bytes objs trace name
1 46.14% 46.14% 221522424 9230101 221522424 9230101 300745 java.lang.String
2 46.12% 92.26% 221442424 9230101 221442424 9230101 300746 char[]
3 7.69% 99.95% 36920408 1 110508448 15 300753 java.lang.Object[]
SITES END
For detailed explanation of the output, refer to "Explanation of the HPROF output file" at :
https://www.ibm.com/support/knowledgecenter/en/SSYKE2_8.0.0/com.ibm.java.aix.80.doc/diag/tools/hprof.html
Tools
To analyze binary format of HPROF output file, refer to:
https://blogs.oracle.com/sundararajan/entry/good_bye_hat_welcome_jhat
or IBM Memory Analyzer Tool (MAT) at:
https://www.ibm.com/support/knowledgecenter/SS3KLZ/com.ibm.java.diagnostics.memory.analyzer.doc/homepage/plugin-homepage-ma.html
https://www.ibm.com/developerworks/java/jdk/tools/memoryanalyzer/
Or any other HPROF analyzer tool of choice.
ACTION
ACTION
ACTION
ACTION
ACTION
ACTION
ACTION
ACTION
ACTION
ACTION
ACTION
ACTION
ACTION
ACTION
ACTION
ACTION
| Document Type: | Instruction |
| Content Type: | Howto |
| Hardware: | all Power |
| Operating System: | all AIX Versions |
| IBM Java: | all Java Versions |
| Author(s): | Rama Tenjarla |
| Reviewer(s): | John Carver |
Was this topic helpful?
Document Information
Modified date:
17 June 2018
UID
isg3T1025034