IBM Java for AIX MustGather: Data collection procedure for slow or hung Java process in UnixFileSystem.getBooleanAttributes() method

A. Download the "pdump" shell script

The pdump shell script can be downloaded from the FTP site:
ftp://ftp.software.ibm.com/aix/tools/debug/pdump.sh

Copy the downloaded file to the AIX system using the filename:
/opt/pdump/pdump.sh

From a command prompt, execute the following command to change the permissions on this file:
# chmod 755 /opt/pdump/pdump.sh

B. Enable the system and the user environment for full core dumps
From a command prompt, while logged in as the root user, execute the following commands to enable full core dumps for the system and process environment.
# chdev -l sys0 -a fullcore=true
# chuser fsize=-1 data=-1 core=-1 USERID

C. Redirect or save standard error (stderr) messages to a file

Commonly used application servers may already save standard out and standard error messages to a log file (e.g., SystemOut.log native_stdout.log, SystemErr.log, native_stderr.log) or to the application log file.

For custom applications, redirect the standard error messages by appending "2>&LOG_FILE" or to redirect both the stdout and stderr to a file append ">LOG_FILE 2>&1".

D. Relogin, then restart your application

Perform the following actions inorder for the changes to take effect:

- Stop the application (and node agent/manager, if applicable)
- Relogin as the USERID used in Step B above
- Confirm that full core is enabled and the new ulimits are in effect:

# ulimit -a
# lsattr -El sys0 | grep -i fullcore

# mkdir -p /TMP_PATH/PMR/MM-DD/trace
# mkdir -p /TMP_PATH/PMR/MM-DD/data
# mkdir -p /TMP_PATH/PMR/MM-DD/iptrace

Enable manual generation of javacores and snap traces by adding the command line option:

-Xdump:java+snap:events=user

To designate a directory for the generated logs, instead of the default directory, add the command line option:

-Xdump:directory=SPECIFIC_PATH

Restart the java application (e.g., node agent/manager) from the USERID new login session. Once the application reaches a steady state, enable truss by running the following command :

# truss -aelfdo > /TMP_PATH/PMR/MM-DD/data/truss.out -p JAVA_PID

To start the iptrace, run the following command:

# startsrc -s iptrace -a "/TMP_PATH/PMR/MM-DD/iptrace"

Immediately once the process hang or slowness can be reproduced, from a command prompt, and while logged in as the root user, execute the following commands to collect the required diagnostic data:

# stopsrc -s iptrace

# cd /TMP_PATH/PMR/MM-DD/data
# gencore JAVA_PID core.dmp
# sleep 5
# kill -3 JAVA_PID
# sleep 5
# kill -3 JAVA_PID
# sleep 5
# kill -3 JAVA_PID

# cd /TMP_PATH/PMR/MM-DD/trace
# trace.sh 10

# cd /TMP_PATH/PMR/MM-DD/data
# /opt/pdump/pdump.sh JAVA_PID

Stop 'truss' that was enabled for the java process.

# prtconf > prtconf.out 2>&1
# lslpp -hac > lslpp-hac.out 2>&1
# errpt -a > errpt-a.out 2>&1

{Copy the application logs to the data directory.}
{Copy the stderr/stdout files to the data directory.}

# JAVA_PATH/jre/bin/jextract START_PATH/core.dmp

Prior to packaging and uploading the data, confirm that the following files have been saved in the "/TMP_PATH/MM-DD/UnsatisfiedLinkError/data directory:

a. Javacore files
b. Snap traces
c. Truss output
d. Trace data
e. pdump data
f. core.dmp.zip (jextracted) or snapcore file (mandatory)
g. AIX commands output
h. standard error, standard output, SystemOut, SystemErr, application logs and any other logs generated.

Packaging the files may simplify the upload of the diagnostic data collected. From the command line and while logged in as the root user, execute the commands:

# cd /TMP_PATH/PMR/MM-DD
# tar -cf - data trace iptrace | gzip -c > PMR.MM-DD.tgz

If you do not correctly replace the Generic Terms in the file name with the actual PMR # and date, there will be a delay in both the review of the data and the resolution of this issue.

Upload the packaged data to IBM secured servers using one of upload options provided on the "IBM Java for AIX MustGather: How to upload diagnostic data and testcases to IBM" web page:

http://www-01.ibm.com/support/docview.wss?uid=isg3T1022619