Troubleshooting
Problem
Your application server terminated without warning or a system core is generated but you are not sure what data to collect. The following MustGather will assist you in collecting the critical data to troubleshoot issues with an IBM WebSphere Application Server crash on the HP-UX platform.
Symptom
System core generated, Java™ Virtual Machine (JVM) process disappears or terminates
Resolving The Problem
Your application server terminated without warning or a system core is generated but you are not sure what data to collect. The following MustGather will assist you in collecting the critical data to troubleshoot issues with an IBM® WebSphere® Application Server crash on the HP-UX platform.
PROCESS THE CORE DUMP
Locate the system core, usually named core, in the profile's root directory. It may also be labeled with a process ID, like core.31415
To process the core dump, you must have GDB (or HP's WDB) installed on your system to proceed. If you don't have this installed, skip down to the section about running pstack.
- /opt/langtools/bin/gdb [PATH_JVM] [CORE_FILE]
[PATH_JVM] is the full path and filename of the Java™ binary.
[CORE_FILE] is the full path and filename to the system core.
For [PATH_JVM], use <install_root>/java/jre/bin/<arch>/java . To determine <arch>, consult the following table:
Architecture of Application Server | Architecture of HP-UX | Path Replacement <arch> |
32-bit | 64-bit Itanium | IA64N |
64-bit | 64-bit Itanium | IA64W |
32-bit | PA-RISC | PA_RISC2.0 |
64-bit | PA-RISC | PA_RISC2.0W |
- Executing install_root/java/jre/bin/java -version reveals the correct architecture and bit-level:
# ./java -version
java version "1.6.0.17"
Java(TM) 2 Runtime Environment, Standard Edition (IBM J6_26 build 1.6.0.17-jinteg_22_nov_2012_01_07-b00 06_Dec_2012_14_39 hpux IA64W (SR4))
Java HotSpot(TM) Server VM (build 20.12-b01-jre1.6.0.17-rc1, mixed mode)
IBM Java ORB build orb626-20121008.00
XML build XL TXE Java 1.0.29
XML build IBM JAXP 1.4.7
XML build XML4J 4.5.19
Once GDB has loaded, issue the packcore command:
- (gdb) packcore
A file, packcore.tar.Z, is produced in the current working directory.
Running Pstack (HP-UX 11.31)
On HP-UX 11.31 and higher, the pstack command is able to pull information from the core dumps produced by a JVM.
Run the command below against the core dump and send in the resulting output file.
pstack [CORE_DUMP] > pstack.out
LOCATE THE hs_err_pid FILE
In the profile root, normally the same location as where the core dump would be found, there should also be a file named hs_err_pid<PID>.log (with the process ID replacing the <PID>). This is a text file that shows the failure information.
RUNNING THE hpjdk_mg.sh SCRIPT
Download and run the script hpjdk_mg.sh without any arguments:
- ./hpjdk_mg.sh
The file hpjdk_mg.out is generated in the current directory and will contain some system information..
LOG FILES
Collect the WebSphere Application Server logs from the JVM instance that crashed:
profile_root/logs/server_name/*
Also gather the server.xml file, which is located in the following directory:
profile_root/config/cells/cell_name/nodes/node_name/servers/server_name/server.xml
Also collect the following system log file:
/var/adm/syslog/syslog.log
SUMMARY OF DATA TO COLLECT
- hs_err_pid*.log
- hpjdk_mg.out
- packcore.tar.Z (produced after running the gdb packcore command)
- pstack.out
- server.xml
- /var/adm/syslog/syslog.log
- Application Server Logs
- SystemOut.log
- SystemErr.log
- native_stdout.log
- native_stderr.log
- Verbose GC data (if not in the native_stdout.log)
OTHER INFORMATION
- How can I check the integrity of the core file?
Some checks to confirm a valid core file before sending data to IBM Support. - The core is invalid if gdb produced an error, such as "core is not a core dump: File format not recognized".
- A valid core will contain a line with the string dld.sl when this command is run:
what [core_file]
[core_file] is the location and filename of the core file. - What are hs_err_pid* files?
These text files are produced when a core is dumped by a process. It can be opened in a text editor and explains details about the current thread and what caused the crash. This is a critical file in debugging crashes.
This is generated in the current working directory, same as the core file. The file is named hs_err_pid[PID].log, where [PID] is the process ID.
The heading, and the exact filename, of this file is also sent to the native_stdout.log.
Sample output:
#
# An unexpected error has been detected by HotSpot
# Virtual Machine:
#
# SIGSEGV (11) at pc=78c3af94, pid=17174, tid=38
#
To diagnose or identify a problem, it is sometimes necessary to provide Technical Support with data and information from your system. In addition, Technical Support might also need to provide you with tools or utilities to be used in problem determination. You can submit files using one of following methods to help speed problem diagnosis:
- IBM Support Assistant (ISA)
- Service Request (SR)
- FTP to the Enhanced Customer Data Repository (ECuRep)
Related Information
[{"Product":{"code":"SSEQTP","label":"WebSphere Application Server"},"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Component":"Crash","Platform":[{"code":"PF010","label":"HP-UX"}],"Version":"9.0.0.0;8.5.5;8.5;8.0;7.0","Edition":"Base;Express;Network Deployment","Line of Business":{"code":"LOB36","label":"IBM Automation"}},{"Product":{"code":"SSNVBF","label":"Runtimes for Java Technology"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Component":"Java SDK","Platform":[{"code":"","label":""}],"Version":"","Edition":"","Line of Business":{"code":"LOB36","label":"IBM Automation"}},{"Product":{"code":"SS7JFU","label":"WebSphere Application Server - Express"},"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Component":null,"Platform":[{"code":"PF010","label":"HP-UX"}],"Version":"7.0;6.1;6.0;5.1","Edition":"","Line of Business":{"code":"LOB45","label":"Automation"}}]