Troubleshooting
Problem
Resolving The Problem
If you are experiencing performance, hang, or high CPU issues with WebSphere Application Server on Linux, this MustGather will assist you in collecting the data necessary to diagnose and resolve the issue. There are two scripts that can be used to collect the performance diagnostic information. Please expand the following section and download one of the scripts and use it to collect information during the problem
Collecting data
Complete the following three steps:
a) If you have not already done so, enable verboseGC and restart the problematic server(s).
CAUTION: In certain Linux systems, using 'kill -3' or the script on the JVM PID may cause a 20-second JVM pause.
./linperf.sh 1234
./linperf.sh 1234 5678 91011
- If the script fails with the errors "Bad Interpreter: No Such File or Directory" or "syntax error near unexpected token `$'do\r'`", please refer to the FAQ section at the bottom for instructions.
- linperf_RESULTS.<DATE>.<TIMESTAMP>.tar.gz (including javacores)
- Server logs (SystemOut.log, SystemErr.log, native_stderr.log, native_stdout.log, and verboseGC data if enabled)
- server.xml for the server(s) that you are providing data for
- /var/log/messages (Linux OS files)
- What is the impact of enabling verboseGC?
verboseGC Data is critical to diagnosing these issues. This can be enabled on production systems because it has a negligible impact on performance (< 2%).
- What is the linperf_RESULTS.tar.gz file and where can I find it?
The linperf_RESULTS.<DATE>.<TIMESTAMP>.tar.gz file is created while running the linperf.sh script and contains output from the commands called by the script. It will be created in the directory from which you execute the script.
- Does the script gather javacores?
Yes if javacores are in the default directory or in the user-specified directory (set by -Xdump:directory generic JVM argument or IBM_JAVACOREDIR environment variable).
- What are 'javacores' and where do I find them?
javacores are snapshots of the JVM activity and are essential to troubleshooting these issues. These files will usually be found in the <profile_root>. If you don't find the files here, you can search your entire system for them using the following command:
find / -name "*javacore*" - The script fails with with the errors "Bad Interpreter: No Such File or Directory" or "syntax error near unexpected token `$'do\r'`" .
The following methods can lead to script fails when used in Windows:
- Copied the script’s content from a browser into a local file and then copied it to a Linux box using FTP or SCP.
- Copied the script’s content directly into a new file on a Linux box.
For details and solutions, see this Fix: “Bad Interpreter: No Such File or Directory” Error in Linux .
Instead of manually fixing the script after doing above actions, this approach is easier and faster to get the job done:
1. Download the script using a browser (or a curl command).
2. Use the following scp command in Command Prompt or PowerShell to copy the script to a Linux box:scp c:/path/to/linperf.sh user_name@<ip_address>:/path/to/put
Or quick way is using both curl and scp in one command line that downloads the script to a local box and then copies it to a Linux box:curl -o linperfh.sh https://www.ibm.com/support/pages/system/files/inline-files/linperf.sh && scp linperf.sh user_name@<ip_address>:/path/to/put
- Monitor process sizes and paging usage (Only if asked to do so)
The linmon.sh script will collect data every 5 minutes until it is stopped manually. Run the following command before the issue occurs to start the script:
./linmon.sh
This will create two files: ps_mon.out and vmstat_mon.out.
To diagnose or identify a problem, it is sometimes necessary to provide Technical Support with data and information from your system. In addition, Technical Support might also need to provide you with tools or utilities to be used in problem determination. You can submit files using one of following methods to help speed problem diagnosis:
- IBM Support Assistant (ISA)
- Service Request (SR)
- FTP to the Enhanced Customer Data Repository (ECuRep)
Previous linperf.sh versions:
20230626_linperf.sh
20230223_linperf.sh
20221122_linperf.sh
20220622_linperf.sh
2019_linperf.sh
Related Information
Was this topic helpful?
Document Information
Modified date:
23 October 2023
UID
swg21115785