Crash on Linux produces no core or truncated core

Technote (troubleshooting)


Problem(Abstract)

This document outlines what needs to be done to ensure that a full core file is produced on Linux if WebSphere Application Server crashes.

Resolving the problem



Follow these directions in order, unless directed by IBM support.

NOTE: The settings below require a restart of your application server. If you also use a nodeagent to start your server(s), you will need to restart this as well. Changing ulimit settings additionally require the restart to occur on the same command line (or terminal) session.

1. Setting Ulimits
See Also: Guidelines for Setting Ulimits

To set ulimits on the core and file sizes to unlimited, run these two commands as the user who starts the nodeagent and/or application server

ulimit -c unlimited
ulimit -f unlimited


You can run ulimit -a to verify current ulimit settings.

Ulimits can also be altered at a global level. See the FAQ for more information.



2. Verifying Hard Disk Space

Check your partitions where WebSphere Application Server resides and make sure there is enough space for the dump to be produced. Usually an error message will be seen in the native_stderr.log that indicates if the core was unable to be written.

To check all of your partitions, execute this command:

df -k






Additional steps if still unable to capture a core file


3. Disable Signal Handlers

Sometimes a loaded library or external process can trap some signals, especially signal 3 and 11, which prevent any core file generation by the JVM.

    a. Disable MQ Signal Traps

    WebSphere MQ is known to trap a subset of signals that the JVM also uses. If you are using WebSphere MQ, or are not sure, simply add this environment variable to your configuration:


    name:  MQS_NO_SYNC_SIGNAL_HANDLING
    value: true



    b. Disable All Signal Handlers

    To force the operating system to handle all signals sent to the JVM process, you can disable all JVM signal handlers.

    For IBM SDK 5.0 and later, set this JVM argument:
    -Xrs

    NOTE: On SDK 6.0, to prevent unintentional crashes due to SIGTRAP, clear the shared class cache by executing <WAS_HOME>/bin/clearClassCache.sh

    For prior versions of the IBM SDK, set this environment variable:
    name:  IBM_NOSIGHANDLER
    value: true  



4. Disable Javacore Generation

On rare instances, disabling javacore generation will help produce a core file.
To disable, simply add the following environment variable:

name:  DISABLE_JAVADUMP
value: true






Frequently Asked Questions (FAQ)

What happens if I do not have write permission in the profile's root directory, or the directory I am redirecting javacores, heapdumps, and system core files to?

This will result in a failure when writing these files to the system. The error may be recorded in the native_stderr.log.



Even with all ulimit settings set to unlimited, core files are truncated at 2GB?

This is a limitation on 32-bit processes. You can avoid this issue if you enable large file support on the operating system, or use a 64-bit version of WebSphere Application Server.

Additionally, running out of free space can cause file truncation.



Can I test my configuration to see if a core can be generated?

Yes you can simulate a crash by sending a signal 6 to the JVM process. This will terminate the process.

kill -6 PID

An alternative is to use the gcore command. This will produce a core file and will allow the process to continue running.

gcore PID


Are ulimit settings permanent?

No, they are temporary and last as long as the session is alive. Ulimits are set on a per user basis, and the settings are applied per session, such as a command-line window. If a brand new session is started, and is not spawned from the current session, the ulimits will load the defaults.



Can I set ulimit settings globally?
By editing the /etc/security/limit.conf file, ulimit settings can be set globally.

However, if the application server is started by the init process at startup, these settings will not take effect. You will need to use the ulimt command line settings directly in the init.d script.



Where can core files be generated?

Normally found in the profile's root directory, but can be in a number of alternative locations. Try searching in these locations first:
  • <WAS_HOME>/profiles/<PROFILE_NAME>
  • <WAS_HOME>/bin/
  • /tmp

If you cannot find a core file in any of these locations, search your entire machine for core* files.

On the IBM SDK, the -Xdump variable can be configured to provide a path for the core dump, for example:
-Xdump:system:file=/path/to/coredump.dmp

Please see the IBM SDK documentation for further information on the -Xdump arguments


The older environment variable IBM_COREDIR is deprecated, but this was also used to configure the core dump path.

Related information

Exchanging information with IBM Technical Support
Steps to get support for WebSphere Application Server
MustGather: Read first for WebSphere Application Server
Troubleshooting Guide for WebSphere Application Server

Cross reference information
Segment Product Component Platform Version Edition
Application Servers WebSphere Application Server - Express Hangs/performance degradation Linux 7.0, 6.1, 6.0, 5.1
Application Servers Runtimes for Java Technology Java SDK

Rate this page:

(0 users)Average rating

Document information


More support for:

WebSphere Application Server
Crash

Software version:

5.1, 6.0, 6.1, 7.0, 8.0, 8.5

Operating system(s):

Linux

Software edition:

Base, Express, Network Deployment

Reference #:

1115658

Modified date:

2008-11-10

Translate my page

Machine Translation

Content navigation