IBM Support

WebSphere Application Server process exits without leaving a footprint in log files or no core dumps

Technote (troubleshooting)


Problem(Abstract)

When running WebSphere Application Server, a Java™ process exits without producing a system core or userdump and no entries are seen in the SystemOut.log in the logs.

Cause

There could be several causes of a Java process exiting without a footprint (no core file produced on UNIX or no user.dmp file produced on Windows) :

  1. The nodeagent has restarted the unresponsive server
  2. Java Native Interface (JNI) code has called JVM to shutdown
  3. Some code within the Java process has called System.exit() or Runtime.exit()
  4. Some external process has triggered a signal that will kill the Java process (SIGTERM .etc)

Resolving the problem

Follow the points below to debug the problem when you suspect that the Java process has exited without leaving any footprint.

Follow the steps in order, however specific OS steps are located at the bottom of the document.

  1. Make sure that the operating system wasn't unintentionally restarted
    Or if a user who owns the running application server process had logged-off the system

    Restarting the operating system or logging off will cause any processes owned by that user to halt operation immediately (services will be gracefully stopped if configured as such). This will make it seem that a crash has occurred without a footprint.

    Check the audit/event logs for any logoff and/or restart activity. If using a virtual machine, check the audit logs for restart activity.

    If running on Windows, see the note at the bottom of the document about Windows Services.

    If #1 is ruled out, continue to the next steps.


  2. Check to see if generic JVM argument -Xrs is configured.
    This option allows the OS signal handlers to handle most process signals and may unintentionally terminate the JVM without a footprint. Removing the argument should return the JVM to normal operation.


  3. Ensure that the nodeagent has not restarted the unresponsive server.
    This is only applicable for Network Deployment installations.

    As per the default monitoring policy the nodeagent will restart any unresponsive server. This may look as if the Java process exited without leaving any footprint. Check the nodeagent logs and see if it has restarted the server.

    You can check the monitoring policy setting through an administrative console page, click Servers > Application Servers > server_name. Then, under Server Infrastructure, click Java and Process Management > Monitoring Policy.


  4. Check if application code has called System.exit()
    Or if application code is triggering a graceful shutdown of the JVM.

    When System.exit() (or similarly Runtime.exit()) is called, this immediately terminates the JVM, without logging what had happened. This will make it seem like a system crash, but without the core dump or javacores that would be associated with such an event. Steps below go into how to trigger javacores on calls to System.exit().

    NOTE: If using a network deployment edition of WebSphere Application Server, the nodeagent may restart the server automatically.

    The next substeps goes into how the SDK can report an unintentional shutdown of the JVM.


    • Enable dumps to be generated upon server exit/shutdown. There are several ways to enable this as each step below will have slightly different results.
      1. Enable a custom property to trigger thread dumps (javacores) on shutdown. It will also log the incident in the SystemErr.log with the current thread that initiated the shutdown, as usually a shutdown hook message will be seen, sometimes with a thread stack:

        EXAMPLE MESSAGE:
        [3/14/15 03:14:15:926 MST] 000000ab SystemErr     R java.lang.Exception: Thread[Thread-3141,5,main]: shutdown hook stack trace

        -Dcom.ibm.ws.runtime.dumpShutdown=true
      2. On AIX, Linux, and Windows, use the IBM SDK’s Xtrace mechanism to trigger a javadump on System.exit() (may optionally add another argument for Runtime.exit())

        -Xtrace:trigger=method{java/lang/System.exit,javadump}

      3. Generic JVM arguments can also be used on AIX, Linux, and Windows with the IBM SDK to trigger javacores and system dumps on vmstop events (when the JVM is stopped).

        -Xdump:java:events=vmstop
        -Xdump:system:events=vmstop


        The above options will generate a javacore and a system dump whenever the Java Virtual Machine stops or a kill signal from the OS is received. Please refer Xdump documentation for more details. Once the system dump is generated (from an IBM SDK), process it with jextract.



  5. Check if Java Native Interface (JNI) code have called JVM to shutdown.
    Verbose JNI trace reports information about use of native methods and other Java Native Interface activity. If the application's native code made a call to "JNI_DestroyJavaVM" or System.exit(), it will be logged.

    The verbose JNI can log all the interactions between the native code and Java. However, if there are issues in the native code, JNI trace will not be all that helpful in this particular case.

    Follow the steps below to enable verbose JNI :
    1. Select Servers
    2. Select Server Types.
    3. Select WebSphere Application Servers > name_of_appserver> Process Definition.
    4. Select Java Virtual Machine.
    5. Place a check mark next to Verbose JNI, then click apply.
    6. At the top of the administrative client, click Save to save the changes to the master configuration.
    7. Restart the application server.

      The verbose JNI output will normally be logged in the native_stderr.log file for the application server, and may produce a lot of output. Clearing out the logs is recommended before you start the server with this setting in place.


  6. HP-UX
    • TUSC
      This is a system trace tool usually available on HP-UX systems similar to truss. The following command below should output the system calls and signals along with timestamps.

      tusc -T "%Y-%m-%dT%H:%M:%S" -f -o /tmp/tusc/output/folder/tusc.out PID

      Note: A more verbose output is available with the -v option.


  7. LINUX
    • OOM KILLER
      The OOM killer can terminate a process without a footprint. This mechanism will normally kill unimportant processes if the system encounters an Out of Memory event, but may end up terminating the application server JVM.

      A system administrator can retrieve the /var/log/messages file to review for evidence of such an event.

      EXAMPLE MESSAGES:
      Mar 14 03:14:15 xxxxx kernel: yyyyy invoked oom-killer: ...
      ...
      Mar 14 03:14:15 xxxxx kernel: Out of Memory: Killed process 31415

    • STRACE
      Collect strace on a running process. This will produce an output file in which the strace data will be stored. The command allows entry of a specific directory where the strace output will be generated.

      strace -tt -f -o /tmp/strace/output/folder/strace.out -p PID


  8. SOLARIS
    • TRUSS
      This utility can track the signals the application server process receives. You can attach truss to an already running process. Start with the command below which will output data with date stamps, and track forked processes.

      truss -d -f -o /tmp/truss/output/folder/truss.out -p PID
    • STRACE
      A version of strace is available for Solaris, but it may not be present on all systems. You can run this against an already running process, and this will produce an output file in which the strace data will be stored. The command allows entry of a specific directory where the strace output will be generated.

      strace -tt -f -o /tmp/strace/output/folder/strace.out -p PID


  9. WINDOWS
Submitting required data
Gather the following files:
  • Server Logs and Native Logs (SystemOut.log, SystemErr.log, native_stdout.log, native_stderr.log)
  • All javacore*.txt files
  • core*.zip, generated by jextract (IBM SDK only)
  • Collect all strace/truss or related data produced
  • /var/log/messages (UNIX/Linux Only)
Note: You can archive or compress the preceding files.

Send the results to IBM support.

Related information

Processing Core Dumps with Jextract (IBM SDK only)


Cross reference information
Segment Product Component Platform Version Edition
Application Servers Runtimes for Java Technology Java SDK

Document information

More support for: WebSphere Application Server
Java SDK

Software version: 7.0, 8.0, 8.5, 8.5.5, 9.0

Operating system(s): AIX, HP-UX, Linux, Solaris, Windows

Software edition: Base, Express, Liberty, Network Deployment

Reference #: 1144595

Modified date: 14 June 2018