IBM Support

MustGather: Performance, hang, or high CPU issues on Solaris

Troubleshooting


Problem


If you are experiencing performance degradation, hang, no response, hung threads, CPU starvation, high CPU utilization, network delays, or deadlocks, this MustGather will assist you in collecting the critical data that is needed to troubleshoot your issue.

Resolving The Problem

If you are experiencing performance degradation, hang, no response, hung threads, CPU starvation, high CPU utilization, network delays, or deadlocks, this MustGather will assist you in collecting the critical data that is needed to troubleshoot your issue.

Complete the following three steps:



(1) Collecting the required data:

If you have not already done so, enable verboseGC and restart the problematic server(s) .

At the time of the problem, run the attached script with the following command:

./solperf.sh [PID]
This script will create a file named solperf_RESULTS.tar.gz and create thread dumps at a interval of 30 seconds. Please note on solaris there will be no separate javacore files but instead they are called thread dumps. About  ten to twelve thread dumps will be written into the native_stdout.log file. This script should be executed as the root user. As with any script, you may need to add execute permissions before executing the script (chmod).

In the above command, [PID] is the Process ID of the problematic JVM(s). If specifying multiple Process IDs they should each be separated by a space.

(2) Collecting log files and configuration data:

Collect the server logs (SystemOut.log, SystemErr.log, native_stderr.log, native_stdout.log) from the problematic server(s):

profile_root /logs/ server_name /*

Collect the server.xml from the problematic server(s):

profile_root /config/cells/ cell_name /nodes/ node_name /servers/ server_name /
server.xml


(3) Finishing up and submitting the results:

Zip/Tar all the files gathered:
  • solperf_RESULTS.tar.gz
  • server logs (SystemOut.log, SystemErr.log, native_stderr.log, native_stdout.log)
  • server.xml

Send the results to IBM Support: "Exchanging information with IBM Support"


Frequently Asked Questions:
  • What is the impact of enabling verboseGC?
    VerboseGC data is critical to diagnosing these issues. This can be enabled on production systems because it has a negligible impact on performance (< 2%).

    What is the solperf_RESULTS.tar.gz file and where can I find it?
    The solperf_RESULTS.tar.gz file is created by the solperf.sh script and contains output from the commands called by the script. It will be created in the directory from which you execute the script.


Related information

Recording your screen to share with IBM Support

Cross reference information
Product Component Platform Version Edition
WebSphere Application Server - Express Hangs/performance degradation Solaris 8.5, 8.0, 7.0, 6.1
Runtimes for Java Technology Java SDK

Document information

More support for: WebSphere Application Server

Component: Hangs/Performance Degradation

Software version: 6.1, 7.0, 8.0, 8.5, 9.0

Operating system(s): Solaris

Software edition: Base, Express, Network Deployment

Reference #: 1115625

Modified date: 18 April 2019