IBM Support

Mustgather: Gathering data for hang or performance problems on Linux

Technote (troubleshooting)


Problem(Abstract)

If your Java application experiences perceived hang or performance problems on a Linux operating system, there are a number of diagnostic data files that are useful for diagnosing the problem. This mustgather document describes about the diagnostic data files to be collected and the procedure to collect the same.

Resolving the problem

The following set of diagnostic data files need to be collected manually during the time of hang or performance problems perceive in your java application:
1. Javacores
2. Core dumps
3. Verbose GC logs (native_stderr.log)
4. Process logs (ps)
5. Paging usage logs (vmstat)
6. CPU Profiling logs(Tprof logs, in case of performance problems)
7. Netstat logs
8. Health Center logs (In case of performance problems)

For these diagnostic data files to be created, check that the system is configured correctly as per setup document.

Please collect the diagnostic data files manually during the time of hang or performance problems perceive in your java application as per the below steps:

1. Generate at least three javacores and core dumps using the following command:
kill -3 [PID_of_problem_JVM]

There needs to be 2 or 3 minutes time interval in between each set of dumps(javacore and core dump) generated. Otherwise spread the time interval across the issue time and generate the above set of diagnostic data files.

Please note that all the javacores and the core dumps generated can be found in the current path and the names of them will follow the below format:
Javacore : javacore.<time stamp>.<id>.txt
Core dump : core.<time stamp>.<id>.dmp

Run the JExtract tool against the generated core dump using the following command:
<installpath>/jre/bin/jextract [coredump]

For 64bit JVM enabled with -Xcompressedrefs option, please use
<installpath>/jre/bin/jextract -J-Xcompressedrefs [coredump]

Save the resulting archive file, which has a name in the format of
core.<time stamp>.<id>.dmp.zip

2. In case if the core dumps are not generated as per above step, another method for generating the core dump is by using Linux gcore utility.

In a separate terminal, run the following command:
gcore <process-id>

This command creates a dump file named:
core.<time stamp>.<id>.dmp

Run the JExtract tool against the generated core dump using the following command:
<installpath>/jre/bin/jextract [coredump]

For 64bit JVM enabled with -Xcompressedrefs option, please use
<installpath>/jre/bin/jextract -J-Xcompressedrefs [coredump]

Save the resulting archive file, which has a name in the format of
core.<time stamp>.<id>.dmp.zip

Rename the core dump generated to something unique before generating the next dump file, otherwise you might lose useful information from the dump file.

3. Collect Verbose Garbage Collector data. This data is in the location specified during setup. Alternatively, the data is sent to the stderr output.

4. Obtain a current listing of the processes running on the system, by using the following command:
ps -efH > ps_efh.txt

5. Collect process size monitoring data. The data is in the location specified during setup.

6. Collect paging size usage data. The data is in the location specified during setup.

7. Run the tprof tool and profile the java application during the time of performance problems and collect the profiling data that are stored in the current path.

8. Collect the netstat log containing the snapshot of network activities, by using the following command:
netstat -an > netstat1.out

Please collect multiple netstat logs during the time of hang or performance problems in your Java application.

9. In case of performance problems, run the Health Center tool and profile the application as specified in setup. Please refer the IBM Health Center document for more information.

After collecting all the above diagnostic data files, you can submit them for help with diagnosing the problem.

Document information

More support for: Runtimes for Java Technology
Hangs/Performance Degradation

Software version: 5.0, 6.0, 6.1, 7.0, 7.1, 8.0

Operating system(s): Linux

Software edition: J2SE

Reference #: 1222466

Modified date: 07 March 2017