IBM Support

MustGather: Performance, hang, or high CPU issues with WebSphere Application Server on AIX

Troubleshooting


Problem

If you are experiencing performance, hang, or high CPU issues with WebSphere Application Server on AIX, this MustGather assists you in collecting the data necessary to diagnose and resolve the issue.

Diagnosing The Problem

Complete the following steps:
(1) Collecting the required data:
a) If verboseGC is not enabled, enable verboseGC and restart the server.

b) Download aixperf.sh from this technote: aixperf.sh
The script creates a file named aixperf_RESULTS.<DATE>.<TIMESTAMP>.tar.gz and ten to twelve javacore text files, and completes in 4 minutes. As with any script, you might need to add execute permissions before executing the script (chmod).
c) At the time of the problem, run the script against the problematic process ID (PID) number.  Examples:

If your current process ID (PID) is 1234, then run the command:
./aixperf.sh 1234

If you have multiple PIDs, that you want to collect data for, then separate them with a space:
./aixperf.sh 1234 5678
 
d) For a high CPU issue, run this command with -c option and specify the CPU threshold value. For example, to monitor PID(s) with a CPU usage of 80%. Replace the CPU_THRESHOLD_VALUE with 80 or little lower (without percentage). It will monitor the CPU usage of provided PID(s) every 30 seconds. If the CPU usage exceeds the threshold, it will automatically collect data and exit.
./aixperf.sh -c CPU_THRESHOLD_VALUE 1234 5678

** Note: The script does not work using the server name, it must be the current process ID (PID) number.  

(2) Submitting required data:

Zip the files gathered:
  • aixperf_RESULTS.<DATE>.<TIMESTAMP>.tar.gz created by aixperf.sh (including javacores)
  • WAS_HOME/profiles/<profile_name>/logs (SystemOut.log, SystemErr.log, native_stderr.log, native_stdout.log, verbosegc)
  • server.xml for the server(s) that you are providing data for

(3) Send the results to IBM Support: "Exchanging information with IBM Support"


Frequently asked questions:
  • What is the impact of enabling verboseGC?
    VerboseGC data is critical to diagnosing these issues. It can be enabled on production systems because it has a negligible impact on performance (< 1%).
     
  • What is the aixperf_RESULTS.<DATE>.<TIMESTAMP>.tar.gz file and where can I find it?
    The aixperf_RESULTS.<DATE>.<TIMESTAMP>.tar.gz file is created while running the aixperf.sh script and contains output from the commands called by the script. It is created in the directory from which you execute the script.
     
  • Does the script gather javacores?
    Yes if javacores are in the default directory or in the user-specified directory (set by -Xdump:directory generic JVM argument or IBM_JAVACOREDIR environment variable). 
     
  • What are 'javacores' and where do I find them?
    Javacores are snapshots of the JVM activity and are essential to troubleshooting these issues. These files usually can be found in the <profile_root>. If you do not find the files here, you can search your entire system for them using the following command:
    find / -name "*javacore*"


Only if asked to do so by Support:
The aixmon.sh script collects data every 5 minutes until it is stopped manually. Run the following command before the issue occurs to start the script:
./aixmon.sh

It creates two files: ps_mon.out and vmstat_mon.out.  The script can be stopped by using Ctrl+C keys. 

aixmon.sh.zip

 
Previous aixperf.sh versions:
For WebSphere Application Server6.0 and earlier:
./aixperf60.sh 23456 (use the actual PID number for your JVM)

Note: Both aixperf60.sh and dbxtrace_aix.sh are created in the same directory and dbx must be installed to diagnose high CPU issues.

aixperf60.sh.zip  |  dbxtrace_aix.sh.zip

These scripts will create a file named aixperf_RESULTS.tar.gz and javacore text files. These scripts need to be executed as the root user. As with any script, you might need to add execute permissions before executing the script (chmod).

[{"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSEQTP","label":"WebSphere Application Server"},"Component":"Hangs\/Performance Degradation","Platform":[{"code":"PF002","label":"AIX"}],"Version":"9.0;8.5;8.0;7.0;6.1;6.0","Edition":"Base;Express;Network Deployment","Line of Business":{"code":"LOB45","label":"Automation"}},{"Product":{"code":"SS7JFU","label":"WebSphere Application Server - Express"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Component":"Hangs\/performance degradation","Platform":[{"code":"PF002","label":"AIX"}],"Version":"7.0;6.1;6.0.2;6.0.1;6.0;5.1","Edition":"","Line of Business":{"code":"LOB45","label":"Automation"}},{"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSNVBF","label":"Runtimes for Java Technology"},"Component":"Java SDK","Platform":[{"code":"PF002","label":"AIX"}],"Version":"All Versions","Edition":"","Line of Business":{"code":"LOB36","label":"IBM Automation"}}]

Document Information

Modified date:
16 March 2023

UID

swg21052641