IBM Java for AIX MustGather: Data collection procedure for Java security issues

To prepare for these data collection procedures, the process environment needs to be configured to save the additional debug information to a log file.

A. Increase the process file size

From a command prompt and while logged in as the root user, execute the following command to set the file size and the data area to unlimited.

# chuser fsize=-1 data=-1 USERID

B. Redirect or save standard error (stderr) messages to a file

Commonly used application servers may already save standard out and standard error messages to a log file (e.g., SystemOut.log native_stdout.log, SystemErr.log, native_stderr.log) or to the application log file.

For custom applications, redirect the standard error messages by appending "2>&LOG_FILE" or to redirect both the stdout and stderr to a file append ">LOG_FILE 2>&1".

C. Relogin, then restart your application

Perform the following actions inorder for the changes to take effect:

- Stop the application

- Relogin as the "USERID" used in Step 1.A

- Restart the application from the new login session

A. Enabling debug options will result in additional data being stored in memory buffers and written to application logs. The process file and data sizes should be increased during the data collection to ensure the data is complete.

If there are multiple processes executed by multiple user ids experiencing the issue, then all preparation steps must be repeated for each id and process.

To confirm the process environment is configured correctly, login using the "USERID" specified in the steps, then run the command:

# ulimit -a

The values for "file" and "data" should show as:

file(blocks) unlimited
data(kbytes) unlimited

B. As an example, to save standard error message to the file /tmp/stderr.log, use a command line syntax similar to:

# java YOUR_APP 2>/tmp/stderr.log

to save both standard out and standard error messages to the file /tmp/out.log, specify:

# java YOUR_APP > /tmp/out.log 2>&1

To confirm the messages are being redirected to the log file, view the contents of the log file.

C. For application server environments such as WebSphere and WebLogic, both the node manager/agent and the application servers must be restarted after the relogin of "USERID".

The following Java security debug options must be added to the Java application command line:

-Djavax.net.debug=all
-Dcom.ibm.security.jgss.debug=all
-Dcom.ibm.security.krb5.Krb5Debug=all

After applying the command line options, the Java process must be restarted for the changes to take effect.

The command line options can be added using the administrative console for application servers, startup scripts for custom applications, or using the command line.

To view the list of Java security debug options, execute the command:

# java -Djavax.net.debug=help YOUR_APP

To enable Java security debug for SSL only, use the Java command line option:

-Djavax.net.debug=ssl

To confirm the Java security options are enabled, messages similar to the ones below should appear in the log file.

-Dcom.ibm.security.jgss.debug=all
...
>>>JGSS Build-Level: JDK 7 20150325
IBMJSSEProvider2 Build-Level: -20131216
[JGSS_DBG_PROV] main IBMJGSSProvider (version 7.0) loaded

-Dcom.ibm.security.krb5.Krb5Debug=all

.....
[KRB_DBG_KDC] KrbDataInputStream:main: >>>KrbDataInputStream: Bytes read: 0000

-Djavax.net.debug=true

Cert Authorities:

[read] MD5 and SHA1 hashes: len = 42
0000: 0d 00 00 26 03 01 02 40 00 1e 06 01 06 02 06 03 ................

For additional information about the Debug options, click the links below:

javax.net.debug

com.ibm.security.jgss.debug

com.ibm.security.krb5.Krb5Debug

A. After the issue has reoccurred or has been recreated, from a command prompt, execute the following commands to collect the required diagnostic data.

# mkdir -p /TMP_PATH/MM-DD/java_security/data

# cd /TMP_PATH/MM-DD/java_security/data

# cp -r /JAVA_PATH/jre/lib/security ./

Generate a javacore (a.k.a., thread dump)

# kill -3 JAVA_PID

# cp /START_PATH/javacore*txt ./

If not sure of the "START_PATH", execute command:

#procwdx JAVA_PID

Copy the standard error, standard output, and application logs to the new directory created above.

B. Confirm the following files and directories have been collected in the temporary directory:

a. /JAVA_PATH/jre/lib/security

b. javacore files (javacore.*.txt)

c. standard error, standard output, and application logs

A. Examples of commands to be executed:

# mkdir -p /large_fs/01-31/java_security/data

# cd /large_fs/01-31/java_security/data

# cp -r /usr/java6/jre/lib/security ./

# ps -ef | grep -i java

rt 3211380 1 0 May 31 - 1109:46 java -Dsrse_property=/rt/pmr/test/classloader/JvmTest-dir/j_test/etc/aib.ini....

# kill -3 3211380

# procwdx 3211380

3211380: /rt/pmr/test/

Check for the generated javacore in the the above directory:

# cd /rt/pmr/test

# ls -l *javacore*
-rw-r--r-- 1 rtstaff 234048 Aug 25 10:54 javacore.20150825.105402.3211380.0010.txt

# cp /var/myapp/javacore*txt ./

B. Confirm all files and directories have been saved to the data directory:

# cp /var/myapp/*.log ./

# ls jre/lib/security/*

# ls javacore.*.txt

# ls *.log *.out *.txt

After all of the files have been collected, from a command prompt, execute the following commands to package the requested files:

# cd /TMP_PATH/MM-DD/java_security

# tar -cf - data | gzip -c > PMR.MM-DD.tgz

Examples of commands to be executed:

# cd /large_fs/01-31/java_security

# tar -cf - data | gzip -c > 12345.678.000.01-31.tgz

Upload the packaged data to IBM secured servers using one of the upload options provided on the "AIX Java for AIX MustGather: How to upload diagnostic data and testcases to IBM" web page:

http://www-01.ibm.com/support/docview.wss?uid=isg3T1022619