IBM Java for AIX MustGather: Data collection procedure for Java JIT issues

A. Confirm JIT is the root cause (RCA) of the issue:

Check the contents of standard output, the application log and/or the generated Javacore file after the crash. Look for the text similar to the following, which indicates issues with JIT library:

{similar text will be listed in the output, log files, and Javacore files}
Module=/JAVA_PATH/jre/bin/libj9jit.so

{similar text will only be listed in the Javacore file}
1XHFLAGS VM flags:000000000005xxxx

{similar text will only be listed in the output and log files}
vmState=0x0005xxxx

where "xxxx" could be any digit and letter.


Sometimes, instead of the JIT library, the JIT compiled method that is being executed at the time of the crash is shown:

1XHEXCPMODULE Inside compiled method: com/sun/tools/javac/comp/Attr.visitMethodDef(Lcom/sun/tools/javac/tree/........


and the VM flags may or mayn't show "5" in the 5th position (from the end):

1XHFLAGS VM flags:0000000000000000



B. Check for specific method failing JIT compilation:

Javacore and stdout, generally, show the method being compiled if any issues during JIT compilation of a specific method. For example, output similar to the following will be listed:

in the Javacore:

1XHEXCPMODULE Compiling method: com/sun/tools/javac/comp/Attr.visitMethodDef(Lcom/sun/tools/javac/tree/.......


in the stdout:
Method_being_compiled=com/sun/tools/javac/comp/Attr.visitMethodDef(Lcom/sun/tools/javac/tree/......


If JIT issuse, continue following the rest of the instructions on this page, as the problem is related to JIT.

If not a JIT issue, follow the instructions in the "IBM Java for AIX MustGather: Data collection procedure for automatically generated Java process core (binary) dumps" article:

http://www-01.ibm.com/support/docview.wss?uid=isg3T1022708

A. For JIT failures, output displayed in the javacore file and stdout, for example, will be as below. In the output , Module should show library as "libj9jit.so". Check the value of "vmState/VM flags" also in the output.

The vmState is the thread-specific state of what was happening in the JVM at the time of the crash. The value for vmState is a hexadecimal number ending in MSSSS, where M is the SDK component and SSSS is component specific code. Make sure the value of "M" is 5. Ignore the values shown for "SSSS".

The vmState of "0x00050000" indicates JIT compiler issues.
....
Unhandled exception
Type=Segmentation error vmState=0x0005ff05
J9Generic_Signal_Number=00000004 Signal_Number=0000000b
Error_Value=00000000 Signal_Code=00000032
....
......
Module=/home/test/sdk/jre/bin/libj9jit.so
Module_base_address=0000007FE29A6000
......

In the javacore text file, The GPINFO section shows vmState as "VM flags:"

Before proceeding with the setup and data collection instructions to collect complete diagnostic data for analysis, click Step 7. Upload Data for instructions to upload any existing latest generated logs and data collected.


To prepare for these data collection procedures, the process environment needs to be configured to collect complete data and to save the additional debug information to a log file.


A. Set the user ulimits and the system attribute:

From a command prompt and while logged in as the root user, execute following commands to:

{set file, data, core file ulimit sizes to unlimited}
# chuser fsize=-1 data=-1 core=-1 USERID

{set the fullcore system attribute to true}
# chdev -l sys0 -a fullcore=true



B. Redirect or save standard error (stderr) messages to a file.

Commonly used application servers may already save standard out and standard error messages to a log file (e.g., SystemOut.log native_stdout.log, SystemErr.log, native_stderr.log) or to the application log file.

For custom applications, redirect the standard error messages by appending "2>&LOG_FILE" or to redirect both the stdout and stderr to a file append ">LOG_FILE 2>&1".



C. Perform the following actions in order for the changes to take effect:

- Stop the application (and node agent/manager, if applicable)

- Relogin as the USERID used in Step 2.A

- Confirm that full core is enabled and the new ulimits are in effect by executing the commands:

# ulimit -a

# lsattr -El sys0 | grep -i fullcore

- Do not restart the application until Step 3. Configure has been completed.

A. Enabling debug options will result in additional data being stored in memory buffers and written to application logs. The process file, data, core file sizes should be increased during the data collection to ensure the data is complete.

If there are multiple processes executed by multiple user ids experiencing the issue, then all preparation steps must be repeated for each id and process.

To confirm the process environment is configured correctly, login using the "USERID" specified in the steps, then run the command:

# ulimit -a

The values for "file", "data" and "coredump" should show:
file(blocks) unlimited
data(kbytes) unlimited
coredump(blocks) unlimited

Confirm "fullcore" system attribute set to true:

# lsattr -El sys0 | grep -i fullcore
fullcore true Enable full CORE dump True



B. As an example, to save standard error message to the file /tmp/stderr.log, use a command line syntax similar to:

# java YOUR_APP 2>/tmp/stderr.log


to save both standard out and standard error messages to the file /tmp/out.log, specify:

# java YOUR_APP > /tmp/out.log 2>&1


To confirm the messages are being redirected to the log file, view the contents of the log file.


C. For application server environments such as WebSphere and WebLogic, both the node manager/agent and the application servers must be restarted after the relogin of "USERID".

A. Set environment variables:

Note: The environment variables set on the AIX command line in the current USERID session are applicable to all the processes started by USERID . To limit to a specific process, set the environment variables in the startup/profile script for the specific Java application.

For the 32-bit java process, logged in as USERID, set the "LDR_CNTRL=MAXDATA" environment variable to increase the total native memory of the java process:

# export LDR_CNTRL=MAXDATA=0xB0000000@DSA



B. Enable JIT compiler verbose logging

If the "Compiling method" or "Method_being_compiled" is listed in the javacore file and/or stdout as mentioned in "Step 1.B Determine", generate verbose logging of JIT compiler to look into JIT compilation issues for that method:

-Xjit:verbose={compileStart|compileEnd}


C. Enable generation of AIX core file, JIT dump, javacore and snap traces for JIT related crashes:

-Xdump:system+jitdump+java+snap:events=gpf+abort,range=1..3


To specify a non-default directory to write the generated logs to, specify:

-Xdump:directory=SPECIFIC_PATH


D. Restart the java application (e.g., node agent/manager) from the USERID new login session.

Alternate ways of setting java command line options:

A. Environment variable can be used to set Java command line options. However, options specified on java command line take precedence over environment variable settings:

# export IBM_JAVA_OPTIONS="
-Xjit:verbose={compileStart|compileEnd} -Xdump:system+jidump+java+snap:events=gpf+abort,range=1..3"


B. To direct javacore, AIX core file,jitdump and snap trace to specific directories instead of the default directory of START_PATH, set environment variables:

# export IBM_JAVACOREDIR=SPECIFIC_PATH
# export IBM_COREDIR=SPECIFIC_PATH

A. Execute the following commands to collect the required diagnostic data.

# mkdir -p /TMP_PATH/MM-DD/JIT/data
# cd /TMP_PATH/MM-DD/JIT/data


B. Recreate the problem. After the logs are generated, collect libraries associated with the AIX core file:

# JAVA_PATH/jre/bin/jextract START_PATH/core*.dmp


If any errors with the above command, execute command:

# snapcore -d START_PATH/core*.dmp START_PATH/java


Generated logs are written to the default directory of START_PATH or the user specified SPECIFIC_PATH.


C. Collect output of AIX commands from the same AIX LPAR:

# errpt -a > errpt-a.out 2>&1
# oslevel -s > oslevel-s.out 2>&1
# prtconf > prtconf.out 2>&1
# lsps -a > lsps-a.out 2>&1
# lslpp -hac > lslpp-hac.out 2>&1
# instfix -i > instfix-i.out 2>&1
# emgr -lv3 > emgr-lv3.out 2>&1
# ps avwwwg > ps-avwwwg.out 2>&1



D. Copy the files generated to the /TMP_PATH/MM-DD/JIT/data directory created in "Step 4.A":

# cp /START_PATH/core* ./
# cp /START_PATH/jitdump* ./
# cp /START_PATH/javacore* ./
# cp /START_PATH/Snap*trc ./

one of:
# cp /START_PATH/core*zip ./
# cp /START_PATH/snap*txt ./

and standard error, standard output, SystemOut, SystemErr, application logs and any other logs generated.

A. Examples of commands to be executed:

# mkdir -p /large_fs/01-31/JIT/data
# cd /large_fs/01-31/JIT/data


B. If not sure of the "START_PATH", execute command:

# ps -ef | grep -i java

rt 3211380 1 0 May 31 - 1109:46 java -Dsrse_property=/rt/pmr/test/classloader/JvmTest-dir/j_test/etc/aib.ini....


# procwdx 3211380

3211380: /rt/pmr/test/


Check for the generated javacore in the the above directory:

# cd /rt/pmr/test

# ls -l *javacore*
-rw-r--r-- 1 rtstaff 234048 Aug 25 10:54 javacore.20150825.105402.3211380.0010.txt


D. Confirm all files and directories have been saved to the data directory:

# cp /var/myapp/*.log ./
# ls javacore* heapdump* Snap*
# ls *.log *.out *.txt

** MANDATORY **


Prior to packaging and uploading, confirm that the following files have been saved in the "/TMP_PATH/MM-DD/JIT/data" directory:
a. Javacore files
b. jitdump log
c. AIX core files
d. libraries associated with core file
e. Snap traces
f. standard error, standard output, SystemOut, SystemErr, application logs and any other logs generated.
g. AIX commands output

After all of the files have been collected, from a command prompt, execute the following commands to package the requested files:

# cd /TMP_PATH/MM-DD/JIT
# tar -cf - data | gzip -c > PMR.MM-DD.tgz

A. Examples of commands to be executed:

# cd /large_fs/01-31/JIT
# tar -cf - data | gzip -c > 12345.678.000.01-31.tgz

Upload the packaged data to IBM secured servers using one of upload options provided in the "IBM Java for AIX MustGather: How to upload diagnostic data and testcases to IBM" web page:

http://www-01.ibm.com/support/docview.wss?uid=isg3T1022619


If this step is reached by clicking "Step 6. Upload Data" in Step 2, at the end of the completion of the existing latest logs/data upload, click Step 2. Prepare Environment to continue with the data collection instructions.