IBM Java for AIX MustGather: Data collection procedure for automatically generated process core (binary) dumps

To prepare for these data collection procedures, the process environment needs to be configured to enable the creation of complete AIX (process) core dumps.

A. Enable the system and the user environment for full core dumps

From a command prompt, and while logged in as the root user, execute the following commands to enable full core dumps for the system and process environment.

# chdev -l sys0 -a fullcore=true

# chuser fsize=-1 data=-1 core=-1 USERID

B. Relogin, then restart your application

Perform the following actions inorder for the changes to take effect:

- Stop the application (and node agent/manager, if applicable)

- Relogin as the USERID used in Step 1.A

- Confirm that full core is enabled and the new ulimits are in effect by executing the commands:

# ulimit -a

# lsattr -El sys0 | grep -i fullcore

- Restart the application (e.g., node agent/manager) from the new login session

A. When the full core dump options are not enabled and the core dumps are uploaded, in most cases, the core dumps will be incomplete or truncated. Not setting these options will prevent the support specialist from analyzing the data and will also delay the resolution of the reported issue.

When using J2E (or J2EE) application servers such as IBM WebSphere or Oracle WebLogic, for the changes to take effect, both the node agent (manager) and the application (manager) servers have to be stopped and restarted (and relogin before restarting).

Examples of commands to be executed:

# chdev -l sys0 -a fullcore=true

# chuser fsize=-1 data=-1 core=-1 wasadmin

The AIX core file is generated into the current working directory of the process. Use the AIX environment variable:

IBM_COREDIR=NEW_PATH

to specify an alternate location for the AIX (process) core dump. Likewise, use the AIX environment variable:

IBM_JAVACOREDIR=NEW_PATH

to specify an alternate location for the javacore.*.txt files.

Both the IBM_COREDIR and IBM_JAVACOREDIR variables have to be configured for the process prior to the it being started (i.e., as part of its startup procedure and the process has to be restarted).

Once the application has failed and a core dump is produced, from a command prompt, and while logged in as the root user, execute the following commands to collect the required diagnostic data:

# mkdir -p /TMP_PATH/PMR/MM-DD-HH/data1
# cd /TMP_PATH/PMR/MM-DD-HH/data1

# prtconf > prtconf.out 2>&1
# lslpp -hac > lslpp-hac.out 2>&1
# instfix -i > instfix-i.out 2>&1
# emgr -lv3 > emgr-lv3.out 2>&1
# svmon -O segment=category -P > svmon-p.out 2>&1
# ipcs -saPrX > ipcs.out 2>&1
# errpt -a > errpt-a.out 2>&1
# JAVA_COMMAND -version > java-version.out 2>&1

{Copy the javacore.*.txt files to the data1 directory.}
{Copy any Snap.*.trc files to the data1 directory.}
{Copy any heapdump.*.phd files to the data1 directory.}
{Copy application log files to the data1 directory.}

# JAVA_PATH/jre/bin/jextract -J-Xms1024M -J-Xmx1024M \
CORE_PATH core.java.dmp.zip

Note: In situations when the jextract command fails, execute the snapcore command instead of the jextract command:

# snapcore -d ${PWD} CORE_PATH JAVA_COMMAND

Examples of commands to be executed:

** Do not copy and paste AS-IS, these are only examples **

# mkdir -p /large_fs/12345.123.000/01-31-10/data1
# cd /large_fs/12345.123.000/01-31-10/data1

# prtconf > prtconf.out 2>&1
# lslpp -hac > lslpp-hac.out 2>&1
# instfix -i > instfix-i.out 2>&1
# emgr -lv3 > emgr-lv3.out 2>&1
# svmon -O segment=category -P > svmon-p.out 2>&1
# ipcs -saPrX > ipcs.out 2>&1
# errpt -a > errpt-a.out 2>&1
# /usr/java7/bin/java -version > java-version.out

# cp /opt/myapp/javacore*txt ./
# cp /opt/myapp/Snap*trc ./
# cp /opt/myapp/heapdump*hpd ./
# cp /opt/myapp/myapp*logs ./

# /usr/java7/jre/bin/jextract -J-Xms1024M -J-Xmx1024M \
/opt/myapp/core.20150131.010101.1234512.001.dmp \
core.java.dmp.zip

or

# snapcore \
/opt/myapp/core.20150131.010101.1234512.001.dmp \
/usr/java7/bin/java

The jextract utility differs from snapcore by collecting platform-specific details about the application such as word size, endianness, data structure layouts, and symbolic information. This information is captured in an XML file which is packaged with the core dump file.

** Mandatory **

Prior to packaging and uploading, confirm that the following files have been saved in the temporary directory:

1. core.java.dmp.zip or snapcore file

2. javacore.*.txt files

3. Application log files

4. All of the *.out files generated in Step 2

5. heapdump.*.phd files (if available)

6. Snap.*.trc files (if available)

Sending incomplete data or data other than the requested data may delay the resolution of the reported issue.

Examples of commands to execute:

** Do not copy and paste AS-IS, these are only examples **

# cd /large_fs/12345.123.000/01-31-10/data1

# ls core*zip snapcore* javacore* *.out

# ls heapdump* Snap*

# ls *.logs

Packaging the files may simplify the upload of the diagnostic data collected. From the command line, and while logged in as the root user, execute the commands:

# cd /TMP_PATH/PMR/MM-DD-HH

# tar -cf - data1 | gzip -c > PMR.MM-DD-HH.tgz

Examples of commands to execute:

** Do not copy and paste AS-IS, these are only examples **

# cd /large_fs/12345.123.000/01-31-10

# tar -cf - data1 | gzip -c > 12345.123.000.01-31-10.tgz

Upload the packaged data or individual files to an IBM secured server using one of upload options provided on the "MustGather: How to upload diagnostic data and testcases to IBM" web page:

http://www-01.ibm.com/support/docview.wss?uid=isg3T1022619