Default behavior for OutOfMemory errors

WebSphere® Application Server supports IBM® Java™ 6 R26 and later versions on supported operating systems. The default behavior has changed.

By default in IBM Java 5 and later, the first four OOM errors for the lifetime of a Java process produce a PHD-formatted heap dump, a Java dump file (javacore), and a snap dump file. By default in IBM Java 6 R26 and later, the first OOM for the lifetime of a Java process produces a PHD-formatted heap dump, a Java dump file, a snap dump file, and an operating system dump—core file on Linux®, AIX®, and IBM i (not to be confused with a javacore), user-mode minidump with full memory on Windows operating systems, and SYSTDUMP on z/OS®. The second, third, and fourth OOM errors produce only a PHD-formatted heap dump and a Java dump file. Therefore, the change in default behavior is an additional system dump on the first OOM error.

A system dump is a superset of a PHD heap dump. A system dump also includes memory contents (strings, primitives, variable names, and so on), thread and frame local information, some native memory information, and more. For more information, go to https://publib.boulder.ibm.com/httpserv/cookbook/Major_Tools-Eclipse_Memory_Analyzer_Tool.html. This added information can solve a larger class of problems, provide more general insight into a running JVM, and ultimately reduce the time that it takes to solve a problem. In IBM Java 5 R12 or earlier and IBM Java 6 R9 or earlier, a system dump had to be post-processed using the jextract tool. With recent versions of IBM Java, however, including the one that ships with this new OOM behavior, a system dump can be directly loaded by a DTFJ-capable tool such as the Memory Analyzer Tool without any post-processing for OOMs (just like a PHD heap dump). For crashes, however, jextract should still be used.

The default configuration can be observed with the -Xdump:what generic JVM argument. For example, running

$WAS/java/bin/java -version
-Xdump:what

produces the following output (some output removed). Note the range option in particular.

# java -version -Xdump:what
Registered dump agents
...
-Xdump:system:
    events=systhrow,
    filter=java/lang/OutOfMemoryError,
    label=... core.&Y&m&d.&H% M% S.% pid.% seq.dmp,
    range=1..1,
    priority=999,
    request=exclusive+compact+prepwalk
...
-Xdump:heap:
    events=systhrow,
    filter=java/lang/OutOfMemoryError,
    label=... heapdump.% Y% m% d.% H% M% S.% pid.% seq.phd,
    range=1..4,
    priority=500,
    request=exclusive+compact+prepwalk,
    opts=PHD
...
-Xdump:java:
    events=systhrow,
    filter=java/lang/OutOfMemoryError,
    label=... javacore.% Y% m% d.% H% M% S.% pid.% seq.txt,
    range=1..4,
    priority=400,
    request=exclusive+preempt
...
-Xdump:snap:
    events=systhrow,
    filter=java/lang/OutOfMemoryError,
    label=... Snap.% Y% m% d.% H% M% S.% pid.% seq.trc,
    range=1..4,
    priority=300,
    request=serial

Note: A blank space was added to this example after each occurrence of the % character to avoid inappropriate conversions of the text in this documentation.

This configuration can be changed using the -Xdump generic JVM arguments:

To revert to the old behavior on OOM, use:

-Xdump:system:none -Xdump:system:events=gpf+abort+traceassert+corruptcache

To remove PHD heap dumps on OOM, use:
```
-Xdump:heap:none
```

To remove PHD heap dumps on OOM and take system dumps on not just the first OOM but on the first four:

-Xdump:heap:none 
-Xdump:system:none 
-Xdump:system:events=gpf+abort+traceassert+corruptcache 
-Xdump:system:events=systhrow,filter=java/lang/OutOfMemoryError,range=1..4,request=exclusive+compact+prepwalk

You must ensure that operating systems are correctly configured to produce untruncated system dumps:

System dumps are written to the location specified by the file parameter in -Xdump (see the previous label attribute in -Xdump:what). You should change this to a dedicated partition with sufficient space. For example:

-Xdump:system:file=/var/dumps/core.% Y% m% d.% H% M% S.% pid.% seq.dmp

Note: A blank space was added to this example after each occurrence of the % character to avoid inappropriate conversions of the text in this documentation.

You might also consider using the -Xdiagnosticscollector generic JVM argument, which will print a warning to native_stderr.log if the JVM detects that ulimits are set incorrectly on startup.