IBM Support

MustGather: Performance, hang, or high CPU issues with WebSphere Application Server on Linux

Troubleshooting


Problem


If you are experiencing performance, hang, or high CPU issues with WebSphere Application Server on Linux, this MustGather will assist you in collecting the data necessary to diagnose and resolve the issue.

Resolving The Problem

If you are experiencing performance, hang, or high CPU issues with WebSphere Application Server on Linux, this MustGather will assist you in collecting the data necessary to diagnose and resolve the issue. There are two scripts that can be used to collect the performance diagnostic information Please expand the following section and download one of the scripts and use it to collect information during the problem

Collecting data

Complete the following three steps:

(1) Collecting the required data:

If you have not already done so, enable verboseGC and restart the problematic server(s).

At the time of the problem, run the attached script with the following command:

./linperf.sh [PID]

linperf.sh

This script will create a file named linperf_RESULTS.tar.gz and and ten to twelve javacore text files. This script should be executed as the root user. As with any script, you may need to add execute permissions before executing the script (chmod).

In the above command, [PID] is the Process ID of the problematic JVM(s). If specifying multiple Process IDs they should each be separated by a space.

(2) Collecting log files:
Collect the server logs (SystemOut.log, SystemErr.log, native_stderr.log, native_stdout.log) from the problematic server(s):

profile_root /logs/ server_name /*

(3) Submitting required data:

Zip/Tar all the files gathered:
  • linperf_RESULTS.tar.gz
  • all javacores (10 to 12) generated in your profileName folder
  • server logs (SystemOut.log, SystemErr.log, native_stderr.log, native_stdout.log)

Send the results to IBM Support: "Exchanging information with IBM Support"




Frequently Asked Questions:
  • What is the impact of enabling verboseGC?
    VerboseGC data is critical to diagnosing these issues. This can be enabled on production systems because it has a negligible impact on performance (< 2%).

    What is the linperf_RESULTS.tar.gz file and where can I find it?
    The linperf_RESULTS.tar.gz file is created while running the linperf.sh script and contains output from the commands called by the script. It will be created in the directory from which you execute the script.

    What are 'javacores' and where do I find them?
    Javacores are snapshots of the JVM activity and are essential to troubleshooting these issues. These files will usually be found in the <profile_root>. If you don't find the files here, you can search your entire system for them using the following command:

    find / -name "*javacore*"


If asked to do so:
The preceding data is used to troubleshoot most of these type of issues; however, in certain situations Support may require additional data. Only collect the following data if asked to do so by IBM Support.
  • A series of system cores
    • Collect a series of system cores by running the following commands:
      Note: These commands require the gdb debugger to be installed.

      gdb install_root/java/jre/bin/java [PID]
      (gdb) generate core1.[PID]
      (gdb) detach
      (gdb) quit

      <wait two minutes>

      gdb install_root/java/jre/bin/java [PID]
      (gdb) generate core2.[PID]
      (gdb) detach
      (gdb) quit

      <wait two minutes>

      gdb install_root/java/jre/bin/java [PID]
      (gdb) generate core3.[PID]
      (gdb) detach
      (gdb) quit

      Creating core files with gdb as described above should not kill the process. If the above 'generate' command does not work then try using the 'gcore' command.

      Process the resulting system core files (core1.[PID], core2.[PID],...) using the instructions in How to process a core dump using jextract on the IBM SDK on Windows, Linux, and AIX

    System core
    • In cases where we cannot collect a series of system cores we can collect a single core file using the following method. However, it must be noted that this will kill the process when collected and the diagnostic value of a series of system cores is much greater than a single core when working with these type of issues. This single core file will need to be processed with jextract as noted in the above section.

      kill -6 [PID]

    Monitor process sizes and paging usage
    • The linmon.sh script will collect data every 5 minutes until it is stopped manually. Run the following command before the issue occurs to start the script:

      ./linmon.sh

      This will create two files: ps_mon.out and vmstat_mon.out.

Exchanging data with IBM Support

To diagnose or identify a problem, it is sometimes necessary to provide Technical Support with data and information from your system. In addition, Technical Support might also need to provide you with tools or utilities to be used in problem determination. You can submit files using one of following methods to help speed problem diagnosis:


Read first and related MustGathers


Related information

Recording your screen to share with IBM Support

Cross reference information
Product Component Platform Version Edition
WebSphere Application Server - Express Linux 8.5, 8.0, 7.0, 6.1, 6.0, 5.1
Runtimes for Java Technology Java SDK

Document information

More support for: WebSphere Application Server

Component: Hangs/Performance Degradation

Software version: 6.1, 7.0, 8.0, 8.5, 9.0.0.0

Operating system(s): Linux

Software edition: Base, Express, Network Deployment

Reference #: 1115785

Modified date: 17 April 2019


Contact and feedback

Need support?