IBM Java for AIX MustGather: Data collection procedure for locking issues with Java applications

To prepare for these data collection procedures, the pdump and perfpmr tools need to be downloaded. The process environment also needs to be configured to enable the creation of complete AIX (process) core dumps.

A. Download the 'pdump' tool

The pdump tool can be downloaded using this anonymous FTP URL:

ftp://ftp.software.ibm.com/aix/tools/debug/pdump.sh

Install pdump by saving the downloaded file as:

/opt/pdump/pdump.sh

From a command prompt, and while logged in as the root user, execute the command:

# chmod 755 /opt/pdump/pdump.sh

B. Download and install 'perfpmr' tool

The perfpmr tool can be downloaded from the FTP site:

ftp://ftp.software.ibm.com/aix/tools/perftools/perfpmr/

From the above location, select the directory for the AIX operating system release being used, then select and download the perf##.tar.Z (e.g., perf71.tar.Z) file. Copy the downloaded file to the AIX system using the filename:

/tmp/perfpmr.tar.Z

From a command prompt, and while logged in as the root user, execute the following commands:

# mkdir -p /opt/perfpmr
# cd /opt/perfpmr
# uncompress -c < /tmp/perf##.tar.Z | tar -xvf -
# ./Install

C. Enable the system and the user environment for full core dumps

From a command prompt, while logged in as the root user, execute the following commands to enable full core dumps for the system and process environment:

# chdev -l sys0 -a fullcore=true
# chuser fsize=-1 data=-1 core=-1 USERID

D. Enable java verbose garbage collection (GC)

Add the Java command line options:

-verbose:gc
-Xverbosegclog:/TMP_PATH/gc.log (e.g., /tmp/gc.log)

to your java command line or startup profile/script. This will require the process to be restarted

E. Redirect or save standard error (stderr) messages to a file

Commonly used application servers may already save standard out and standard error messages to a log file (e.g., SystemOut.log native_stdout.log, SystemErr.log, native_stderr.log) or to the application log file.

For custom applications, redirect the standard error messages by appending "2>&LOG_FILE" or to redirect both the stdout and stderr to a file append ">LOG_FILE 2>&1".

F. Relogin and restart your application

Perform the following actions inorder for the changes to take effect:

- Stop the application (and node agent/manager, if applicable)

- Relogin as the USERID used in Step 1.C

- Confirm that full core is enabled and the new ulimits are in effect by issuing the commands:

# ulimit -a
# lsattr -El sys0 | grep -i fullcore

- Restart the application (ex. node agent/manager) from the new login session

A. The "pdump.sh" script runs the AIX kernel debugger (kdb) to obtain kernel level data about the process. For a process when several hundreds threads, it may take a few minutes for the script to complete the inspection of all the threads. Because pdump.sh scans active threads, do not run this script for high CPU situations unless instructed to do so.


B. The "perfpmr" data to help us understand and identify the hot spots with AIX, networking, locking, JVM, application, etc. It will collect configuration data as well as run the majority of the AIX performance tools for the duration specified.


C. When the full core dump options are not enabled and the core dumps are uploaded, in most cases, the core dumps will be incomplete or truncated. Not setting these options will prevent the support specialist from analyzing the data and will also delay the resolution of the reported issue.

When using J2E (or J2EE) application servers such as IBM WebSphere or Oracle WebLogic, for the changes to take effect, both the node agent (manager) and the application (manager) servers have to be stopped and restarted.

Examples of commands to be executed:

# chdev -l sys0 -a fullcore=true

# chuser fsize=-1 data=-1 core=-1 wasadmin


The AIX core file is generated into the current working directory of the process. Use the AIX environment variable:

IBM_COREDIR=NEW_PATH

to specify an alternate location for the AIX (process) core dump. Likewise, use the AIX environment variable:

IBM_JAVACOREDIR=NEW_PATH

to specify an alternate location for the javacore.*.txt files.

Both the IBM_COREDIR and IBM_JAVACORDIR variables have to be configured for the process prior to it being started (i.e., as part of its startup procedure and teh process has to be restarted)

A. Immediately once the application exhibits slow performance, from a command prompt and while logged in as the root user, execute the following commands to collect the required diagnostic data:

# mkdir -p /TMP_PATH/PMR/MM-DD/data1
# cd /TMP_PATH/PMR/MM-DD/data1

# /opt/pdump/pdump.sh JAVA_PID

# mkdir -p /TMP_PATH/PMR/MM-DD/perfpmr
# cd /TMP_PATH/PMR#/MM-DD/perfpmr

# perfpmr.sh 240

# cd /TMP_PATH/PMR#/MM-DD/data1

# gencore JAVA_PID ../core.dmp

# kill -3 JAVA_PID
# sleep 5
# kill -3 JAVA_PID
# sleep 5
# kill -3 JAVA_PID

# prtconf > prtconf.out 2>&1
# lslpp -hac > lslpp-hac.out 2>&1
# instfix -i > instfix-i.out 2>&1
# emgr -lv3 > emgr-lv3.out 2>&1
# svmon -O segment=category -P > svmon-p.out 2>&1
# ipcs -saPrX > ipcs.out 2>&1
# errpt -a > errpt-a.out 2>&1
# JAVA_COMMAND -version > java-version.out 2>&1
# cp /TMP_PATH/gc.log ./

{Copy the javacore.*.txt files to the data1 directory.}
{Copy the *.out files to the data1 directory.}
{Copy the application logs to the data1 directory.}
{Copy the stderr/stdout files to the data1 directory.}
{Copy GC log from Step 1.D}

# JAVA_PATH/jre/bin/jextract -J-Xms1024M -J-Xmx1024M \
../core.dmp core.java.dmp.zip

** In situations when the jextract command fails, execute the command:

# snapcore -d ${PWD} ../core.dmp JAVA_COMMAND

A. Examples of commands to be executed:

** Do not copy and paste AS-IS, these are only examples **

# mkdir -p /large_fs/12345.123.000/01-31-10/data1
# cd /large_fs/12345.123.000/01-31-10/data1
#/opt/pdump/pdump.sh 9041

(run perfpmr script for 240 seconds)

# perfpmr.sh 240

(gencore creates am AIX A(process) core file for a running process)

# gencore 1095 ../core.1095

(the kill command generates a Java thread dump)
(in most situations it DOES NOT kill the process)

# kill -3 1095
# sleep 5
# kill -3 1095
# sleep 5
# kill -3 1095

# prtconf > prtconf.out 2>&1
# lslpp -hac > lslpp-hac.out 2>&1
# instfix -i > instfix-i.out 2>&1
# emgr -lv3 > emgr-lv3.out 2>&1
# svmon -O segment=category -P > svmon-p.out 2>&1
# ipcs -saPrX > ipcs.out 2>&1
# errpt -a > errpt-a.out 2>&1
# /usr/java7/bin/java -version > java-version.out
# cp /opt/myapp/javacore*txt ./
# cp /opt/myapp/*.out ./
# cp /opt/myapp/myapp*logs ./

# /usr/java7/jre/bin/jextract -J-Xms1024M -J-Xmx1024M \
../core.dmp core.java.dmp.zip

or

# snapcore ../core.dmp /usr/java7/bin/java

The jextract utility differs from snapcore by collecting platform-specific details about the application such as word size, endianness, data structure layouts, and symbolic information. This information is captured in an XML file which is packaged with the core dump file.

** Mandatory **

Prior to packaging and uploading, confirm that the following files have been saved in the temporary directory:

1. core.java.dmp.zip or snapcore file

2. (three) javacore.*.txt files

3. Application log files

4. All of the *.out files generated in Step 2

5. pdump data

6. perfpmr data

7. Verbose garbage collection log (gc.log)

8. Application log files

A. Sending incomplete data or data other than the requested data may delay the resolution of the reported issue.

Examples of commands to execute:

** Do not copy and paste AS-IS, these are only examples **

# cd /large_fs/12345.123.000/01-31-10/data1
# ls core*zip snapcore* javacore* *.out
# ls *.logs

# ls *.out

# ls gc.log

# ls config.sum monitor.int

A. Packaging the files may simplify the upload of the diagnostic data collected. From the command line and while logged in as the root user, execute the commands:

If you do not correctly replace the Generic Terms in the file name with the actual PMR # and date, the will be a delay in both the review of the data and the resolution of this issue.


# cd /TMP_PATH/PMR/MM-DD
# tar -cf - data1 perfpmr | gzip -c > PMR.MM-DD.tgz

A. Examples of commands to execute:

** Do not copy and paste AS-IS, these are only examples **

# cd /large_fs/12345.123.000/01-31-10
# tar -cf - data1 perfpmr | gzip -c > 12345.123.000.01-31-10.tgz

Upload the packaged data to IBM secured servers using one of upload options provided on the "MustGather: How to upload diagnostic data and testcases to IBM" web page:

http://www-01.ibm.com/support/docview.wss?uid=isg3T1022619