IBM Support

MustGather: Performance, hang, or high CPU issues on Solaris

Troubleshooting


Problem

If you are experiencing performance degradation, hang, no response, hung threads, CPU starvation, high CPU utilization, network delays, or deadlocks, this MustGather will assist you in collecting the critical data that is needed to troubleshoot your issue.

Resolving The Problem

If you are experiencing performance degradation, hang, no response, hung threads, CPU starvation, high CPU utilization, network delays, or deadlocks, this MustGather will assist you in collecting the critical data that is needed to troubleshoot your issue.

Complete the following three steps:



(1) Collecting the required data:

If you have not already done so, enable verboseGC and restart the problematic server(s).

At the time of the problem, run the attached script with the following command:

./solperf.sh [PID]
This script will create a file named solperf_RESULTS.tar.gz and create thread dumps at a interval of 30 seconds. Please note on solaris there will be no separate javacore files but instead they are called thread dumps. About ten to twelve thread dumps will be written into the native_stdout.log file. This script should be executed as the root user. As with any script, you may need to add execute permissions before executing the script (chmod).

In the above command, [PID] is the Process ID of the problematic JVM(s). If specifying multiple Process IDs they should each be separated by a space.

(2) Collecting log files and configuration data:

Collect the server logs (SystemOut.log, SystemErr.log, native_stderr.log, native_stdout.log) from the problematic server(s):

profile_root/logs/server_name/*

Collect the server.xml from the problematic server(s):

profile_root/config/cells/cell_name/nodes/node_name/servers/server_name/
server.xml


(3) Finishing up and submitting the results:

Zip/Tar all the files gathered:
  • solperf_RESULTS.tar.gz
  • server logs (SystemOut.log, SystemErr.log, native_stderr.log, native_stdout.log)
  • server.xml

Send the results to IBM Support: "Exchanging information with IBM Support"


Frequently Asked Questions:
  • What is the impact of enabling verboseGC?
    VerboseGC data is critical to diagnosing these issues. This can be enabled on production systems because it has a negligible impact on performance (< 2%).

    What is the solperf_RESULTS.tar.gz file and where can I find it?
    The solperf_RESULTS.tar.gz file is created by the solperf.sh script and contains output from the commands called by the script. It will be created in the directory from which you execute the script.

[{"Product":{"code":"SSEQTP","label":"WebSphere Application Server"},"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Component":"Hangs\/Performance Degradation","Platform":[{"code":"PF027","label":"Solaris"}],"Version":"9.0;8.5;8.0;7.0;6.1","Edition":"Base;Express;Network Deployment","Line of Business":{"code":"LOB36","label":"IBM Automation"}},{"Product":{"code":"SS7JFU","label":"WebSphere Application Server - Express"},"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Component":"Hangs\/performance degradation","Platform":[{"code":"PF027","label":"Solaris"}],"Version":"8.5;8.0;7.0;6.1","Edition":"","Line of Business":{"code":"LOB45","label":"Automation"}},{"Product":{"code":"SSNVBF","label":"Runtimes for Java Technology"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Component":"Java SDK","Platform":[{"code":"","label":""}],"Version":"","Edition":"","Line of Business":{"code":"LOB36","label":"IBM Automation"}}]

Document Information

Modified date:
18 April 2019

UID

swg21115625