Troubleshooting
Problem
Resolving The Problem
Your application server terminated without warning or a system core is generated but you are not sure what data to collect. The following MustGather will assist you in collecting the critical data to troubleshoot issues with an IBM® WebSphere® Application Server crash on the Linux® platform.
DATA TO COLLECT:
WebSphere Application Server
Location of dumps are usually in the profile root (/opt/IBM/WebSphere/AppServer/profiles/PROFILE_NAME/)
- javacore*.txt, heapdump*.phd, snap*.trc files (if produced)
- jextract output (from processing the IBM SDK dump file)
- Provide the core dump if processing failed
- Appserver logs, including the following:
- SystemOut.log
- SystemErr.log
- native_stderr.log
- native_stdout.log
- /var/log/messages (Linux OS files)
- Configuration data: server.xml
- NOTE: You may also run the collector tool to capture all configuration (and logs) for a particular profile into a single JAR file. The collector.sh script is located in the bin directory in the profile’s root.
WebSphere Liberty
Location of dumps are usually in the server root (/opt/IBM/wlp/user/servers/SERVER_NAME/)
- javacore*.txt, heapdump*.phd, snap*.trc files (if produced)
- jextract/jpackcore output (from processing the IBM SDK dump file) or jstack output (latter on Oracle JDK)
- Provide the core dump if processing failed or if you had run jstack (latter on Oracle JDK)
Also on Oracle JDK, hs_err_pid file will be generated
- Provide the core dump if processing failed or if you had run jstack (latter on Oracle JDK)
- Appserver logs, including
- console.log
- messages.log
- Verbose GC data (if verbose GC is turned on and GC data is redirected to a file)
- /var/log/messages
- Configuration data, including
- server.xml
- jvm.options
PROCESSING THE CORE DUMP
IBM SDK (WebSphere Application Server and WebSphere Liberty)
http://www.ibm.com/support/docview.wss?uid=swg21577379
Make sure to attempt processing the core dump with the command jextract. You will need to locate which IBM SDK was used to run the application server (the javacore contains this information, or simply restart your application server and locate the process with the ps command), and run the associated jextract executable for that SDK install.
Example with a 8192 MB heap, and a JVM that was running with compressed references:
./jextract -J-Xcompressedrefs -J-Xmx8192m /opt/IBM/WebSphere/AppServer/profiles/Dmgr01/core.DATE.TIME.PID.COUNT.dmp
The result will be a zip file that contains the core dump, some XMLs, and any required libraries for processing.
Oracle® JDK (WebSphere Liberty)
If you happen to be running an Oracle JDK on WebSphere Liberty and have a crash, the core dump can be processed with the jstack command. This will pull all of the threads, and you can redirect the output to a text file..
http://docs.oracle.com/javase/7/docs/technotes/tools/share/jstack.html
Syntax
./jstack [options] JAVA_EXE CORE_FILE > jstack.out
JAVA_EXE is the path + filename of the java executable. You can determine this from the output of the ps command when the process is normally running.
For other options, see the link above, but to name a few:
- Use the -m option so both Java and native stack frames are displayed.
- Use the -l option to print out locks.
- The option -J-d64 may be required for 64-bit JVMs.
In addition, an hs_err_pid file will be generated in the profile root (with the process id appended to the end of the filename).
NO CORE DUMP GENERATED
If you find no core dump, and none of the logs (such as native_stderr.log) indicate one was generated, check first if the nodeagent had terminated the JVM and restarted it, as you may not get a dump if that occurs.
Another reason for this, especially if a crash was registered in the error output, is that the ulimits for core and file are not set to unlimited. See Crash on Linux produces no core or truncated core for more information.
If there were no dumps and no signs of a crash, but the process did terminate, check this document for further debugging.
WebSphere Application Server process exits without leaving a footprint in log files or no core dumps
http://www.IBM.com/support/docview.wss?uid=swg21144595
OTHER SCRIPTS
If jextract continues to fail, and you have ensured that a full core dump is processing, you can try these alternative scripts to attempt at pulling the correct libraries so IBM support can utilize the dump. This should only be done if requested by IBM support.
All scripts require the GNU Debugger (GDB) is installed on the system.
gdb -x gdb_commands.txt [JAVA_PATH] [CORE_FILE] > gdb_out.txt
./libsgrabber.sh [JAVA_PATH] [CORE_FILE]
This generates the file libs.tar.gz in the current directory.
[CORE_FILE] should contain the complete path of the core file.
[JAVA_PATH] is the location to the java executable. Replace with install_root/java/jre/bin/java
RELATED INFORMATION
How to process an IBM SDK core dump with jextract
Crash on Linux produces no core or truncated core
Steps to getting support for WebSphere Application Server
Exchanging information with IBM Technical Support
MustGather: Read first for WebSphere Application Server
Troubleshooting guide for WebSphere Application Server
To diagnose or identify a problem, it is sometimes necessary to provide Technical Support with data and information from your system. In addition, Technical Support might also need to provide you with tools or utilities to be used in problem determination. You can submit files using one of following methods to help speed problem diagnosis:
- IBM Support Assistant (ISA)
- Service Request (SR)
- FTP to the Enhanced Customer Data Repository (ECuRep)
Related Information
Was this topic helpful?
Document Information
Modified date:
01 February 2024
UID
swg21104706