IBM Java for AIX MustGather: Data collection procedure for Java heap issues

Before proceeding with the setup and data collection instructions to collect complete diagnostic data for analysis, click Step 6. Upload Data for instructions to upload any existing generated logs and data collected.

To prepare for these data collection procedures, the process environment needs to be configured to collect complete data and save the additional debug information to a log file.


A. Set the user ulimits and the system attribute:

From a command prompt and while logged in as the root user, execute following commands to:

{ set file, data, core file ulimit sizes to unlimited }
# chuser fsize=-1 data=-1 USERID

{ set the fullcore system attribute to true }
# chdev -l sys0 -a fullcore=true

B. Redirect or save standard error (stderr) messages to a file

Commonly used application servers may already save standard out and standard error messages to a log file (e.g., SystemOut.log native_stdout.log, SystemErr.log, native_stderr.log) or to the application log file.

For custom applications, redirect the standard error messages by appending "2>&LOG_FILE" or to redirect both the stdout and stderr to a file append ">LOG_FILE 2>&1".

C. Perform the following actions in order for the changes to take effect:

- Stop the application

- Relogin as the "USERID" used in Step 1.A

- Confirm that full core is enabled and the new ulimits are in effect by executing the commands:

# ulimit -a

# lsattr -El sys0 | grep -i fullcore


- Do not restart the application until Step 2. Configure has been completed

A. Enabling debug options will result in additional data being stored in memory buffers and written to application logs. The process file and data sizes should be increased during the data collection to ensure the data is complete.

If there are multiple processes executed by multiple user ids experiencing the issue, then all preparation steps must be repeated for each id and process.

To confirm the process environment is configured correctly, login using the USERID specified in the steps, then run the command:

# ulimit -a

The values for "file" and "data" should show as:
file(blocks) unlimited
data(kbytes) unlimited

B. As an example, to save standard error message to the file /tmp/stderr.log, use a command line syntax similar to:

# java YOUR_APP 2>/tmp/stderr.log

to save both standard out and standard error messages to the file /tmp/out.log, specify:

# java YOUR_APP > /tmp/out.log 2>&1

To confirm the messages are being redirected to the log file, view the contents of the log file.

A. Set environment variables:

Note: Environment variables set on the AIX command line in the USERID environment are applicable to all the processes started by the user in that environment. To limit to a specific process, set the environment variables in the startup/profile script for that specific java application.

For the 32-bit java process, logged in as USERID, set the "LDR_CNTRL=MAXDATA" environment variable to increase the total native memory of the java process:

# export LDR_CNTRL=MAXDATA=0xB0000000@DSA



B. Add java command line options:

Enable Java verbose garbage collection (GC):

-verbose:gc


By default, verbose GC logging is written to stderr.

If unable or unsure how to redirect standard error for the verbose GC data, specify log file name:

-Xverbosegclog:SPECIFIC_PATH/gc.log

Enable manual generation of javacores, heapdumps and snap traces:

-Xdump:java+heap+snap:events=user


For Out Of Memory issues, generally, Javacores, heapdumps and Snap traces should be generated automatically. To make sure the Javacore, Heapdump and Snap trace files are always generated for Java heap Out Of Memory, add:

-Xdump:java+heap+snap:events=systhrow,filter=java//OutOfMemoryError,range=1..3



To designate a directory for the generated logs, instead of the default directory, add:

-Xdump:directory=SPECIFIC_PATH


C. Restart the java application (e.g., node agent/manager) from the USERID new login session.

B. Alternate ways of setting java command line options and non-default directory for generated log files:

Environment variable can be used to set Java command line options. However, options specified on java command line take precedence over environment variable settings:

# export IBM_JAVA_OPTIONS="-verbose:gc -Xverbosegclog:SPECIFIC_PATH/gc.log -Xdump:java+heap+snap:events=systhrow,filter=java/lan/OutOfMemoryError,range=1..3"



To direct javacore, heapdump and snap trace to specific directories instead of the default directory of START_PATH, set environment variables:

# export IBM_JAVACOREDIR=SPECIFIC_PATH
# export IBM_HEAPDUMPDIR=SPECIFIC_PATH
# export IBM_COREDIR=SPECIFIC_PATH



C. For application server environments such as WebSphere and WebLogic, both the node manager/agent and the application servers must be restarted after the relogin of "USERID".

A. Execute the following commands to collect the required diagnostic data.

# mkdir -p /TMP_PATH/MM-DD/java-heap-issues/data
# cd /TMP_PATH/MM-DD/java-heap-issues/data

B. At the time of the Java heap issues, execute command to generate logs:

# kill -3 JAVA_PID


Generated logs are written to the default directory of "START_PATH" or the user specified SPECIFIC_PATH.


C. Collect output of AIX commands from the same AIX LPAR:

# errpt -a > errpt-a.out 2>&1
# oslevel -s > oslevel-s.out 2>&1
# prtconf > prtconf.out 2>&1
# lsps -a > lsps-a.out 2>&1
# lslpp -hac > lslpp-hac.out 2>&1
# instfix -i > instfix-i.out 2>&1
# emgr -lv3 > emgr-lv3.out 2>&1
# ps avwwwg > ps-avwwwg.out 2>&1


D. Copy the files generated to the data directory "/TMP_PATH/MM-DD/java-heap-issues/data" directory created in step 3.A:

# cp /START_PATH/javacore*txt ./
# cp /START_PATH/heapdump* ./
# cp /START_PATH/Snap*trc ./

Also copy standard error, standard output, SystemOut, SystemErr, gc.log, application logs and any other logs generated to /TMP_PATH/MM-DD/java-heap-issues/data directory.

A. Examples of commands to be executed:

# mkdir -p /large_fs/01-31/java-heap-issues/data

# cd /large_fs/01-31/java-heap-issues/data

D. If "START_PATH" can not easily be determined, execute these commands to help determine the correct path:

# ps -ef | grep -i java

rt 3211380 1 0 May 31 - 1109:46 java -Dsrse_property=/rt/pmr/test/classloader/JvmTest-dir/j_test/etc/aib.ini....

# kill -3 3211380

# procwdx 3211380

3211380: /rt/pmr/test/

Check for the generated javacore in the the above directory:

# cd /rt/pmr/test

# ls -l *javacore*

-rw-r--r-- 1 rtstaff 234048 Aug 25 10:54 javacore.20150825.105402.3211380.0010.txt


Confirm all files and directories have been saved to the data directory:

# cp /var/myapp/*.log ./

# ls javacore* heapdump* Snap*

# ls *.log *.out *.txt

** MANDATORY **


Prior to packaging and uploading the data, confirm that the following files have been saved in the "/TMP_PATH/MM-DD/java-heap-issues/data directory:

a. Javacore files
b. Heapdump files
c. Snap traces
d. verbose GC log
e. standard error, standard output, SystemOut, SystemErr, application logs and any other logs generated.
f. AIX commands output

After all of the files have been collected, from a command prompt, execute the following commands to package the requested files:

# cd /TMP_PATH/MM-DD/java-heap-issues
# tar -cf - data | gzip -c > PMR.MM-DD.tgz

A. Examples of commands to be executed:

# cd /large_fs/01-31/java-heap-issues
# tar -cf - data | gzip -c > 12345.678.000.01-31.tgz

Upload the packaged data to IBM secured servers using one of upload options provided on the "IBM Java for AIX MustGather: How to upload diagnostic data and testcases to IBM" web page:

http://www-01.ibm.com/support/docview.wss?uid=isg3T1022619

If this step is reached by clicking "Step 6. Upload Data" in Step 1, at the end of the completion of the existing latest logs/data upload, click Step 1. Prepare Environment to continue with the data collection instructions.