IBM Support

PM45011: APPLICATION SERVER HANG CAN OCCUR IF A THREAD LOGGING AN FFDC HANGS

Fixes are available

8.0.0.3: WebSphere Application Server V8.0 Fix Pack 3
7.0.0.23: WebSphere Application Server V7.0 Fix Pack 23
8.0.0.4: WebSphere Application Server V8.0 Fix Pack 4
7.0.0.25: WebSphere Application Server V7.0 Fix Pack 25
8.0.0.5: WebSphere Application Server V8.0 Fix Pack 5
7.0.0.27: WebSphere Application Server V7.0 Fix Pack 27
8.0.0.6: WebSphere Application Server V8.0 Fix Pack 6
7.0.0.29: WebSphere Application Server V7.0 Fix Pack 29
8.0.0.7: WebSphere Application Server V8.0 Fix Pack 7
8.0.0.8: WebSphere Application Server V8.0 Fix Pack 8
7.0.0.31: WebSphere Application Server V7.0 Fix Pack 31
7.0.0.27: Java SDK 1.6 SR13 FP2 Cumulative Fix for WebSphere Application Server
7.0.0.33: WebSphere Application Server V7.0 Fix Pack 33
8.0.0.9: WebSphere Application Server V8.0 Fix Pack 9
7.0.0.35: WebSphere Application Server V7.0 Fix Pack 35
8.0.0.10: WebSphere Application Server V8.0 Fix Pack 10
7.0.0.37: WebSphere Application Server V7.0 Fix Pack 37
8.0.0.11: WebSphere Application Server V8.0 Fix Pack 11
7.0.0.39: WebSphere Application Server V7.0 Fix Pack 39
8.0.0.12: WebSphere Application Server V8.0 Fix Pack 12
7.0.0.41: WebSphere Application Server V7.0 Fix Pack 41
8.0.0.13: WebSphere Application Server V8.0 Fix Pack 13
7.0.0.43: WebSphere Application Server V7.0 Fix Pack 43
8.0.0.14: WebSphere Application Server V8.0 Fix Pack 14
7.0.0.45: WebSphere Application Server V7.0 Fix Pack 45
8.0.0.15: WebSphere Application Server V8.0 Fix Pack 15
7.0.0.23: Java SDK 1.6 SR10 FP1 Cumulative Fix for WebSphere
7.0.0.25: Java SDK 1.6 SR11 Cumulative Fix for WebSphere Application Server
7.0.0.27: Java SDK 1.6 SR12 Cumulative Fix for WebSphere Application Server
7.0.0.29: Java SDK 1.6 SR13 FP2 Cumulative Fix for WebSphere Application Server
7.0.0.45: Java SDK 1.6 SR16 FP60 Cumulative Fix for WebSphere Application Server
7.0.0.31: Java SDK 1.6 SR15 Cumulative Fix for WebSphere Application Server
7.0.0.35: Java SDK 1.6 SR16 FP1 Cumulative Fix for WebSphere Application Server
7.0.0.37: Java SDK 1.6 SR16 FP3 Cumulative Fix for WebSphere Application Server
7.0.0.39: Java SDK 1.6 SR16 FP7 Cumulative Fix for WebSphere Application Server
7.0.0.41: Java SDK 1.6 SR16 FP20 Cumulative Fix for WebSphere Application Server
7.0.0.43: Java SDK 1.6 SR16 FP41 Cumulative Fix for WebSphere Application Server

Subscribe

You can track all active APARs for this component.

APAR status

  • Closed as program error.

Error description

  • An application server JVM can hang if a thread logging an FFDC
    hangs.
    
    The problem can be detected by observing the WSVR0605W hung
    thread notifications in the SystemOut.log.
    
    One thread will be running the FFDCFilter.processException()
    which can be invoked from any WebSphere Application Server
    component.  The thread will get stuck doing its work.
    In the following example, the thread which invoked
    FFDCFilter.processException() is stuck waiting on a response to
    the OracleConnectionWrapper.getMetaData() method call:
    
    [05/24/12 12:06:35:286 EST] 00000016 ThreadMonitor W
    WSVR0605W: Thread "ORB.thread.pool : 32" (000000d6) has
    been active for 304721 milliseconds and may be hung.  There
    is/are 16 thread(s) in total in the server that may be hung.
     at
    oracle.jdbc.OracleConnectionWrapper.getMetaData(OracleConnection
    Wrapper.java:150)
     at
    com.ibm.ws.rsadapter.spi.WSRdbManagedConnectionImpl.introspectSe
    lf(WSRdbManagedConnectionImpl.java:1480)
     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
     at
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessor
    Impl.java:60)
     at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethod
    AccessorImpl.java:37)
     at java.lang.reflect.Method.invoke(Method.java:611)
     at
    com.ibm.ws.ffdc.impl.Formatters$IntrospectSelfFormatter$1.run(Fo
    rmatters.java:99)
     at
    com.ibm.ws.ffdc.impl.Formatters$IntrospectSelfFormatter$1.run(Fo
    rmatters.java:96)
    
    ...
    
     at
    com.ibm.ffdc.util.provider.IncidentEntry.log(IncidentEntry.java:
    96)
     at com.ibm.ffdc.util.provider.Ffdc.log(Ffdc.java:90)
     at
    com.ibm.ws.ffdc.FFDCFilter.processException(FFDCFilter.java:114)
     at
    com.ibm.ejs.j2c.FreePool.createOrWaitForConnection(FreePool.java
    :1671)
     at com.ibm.ejs.j2c.PoolManager.reserve(PoolManager.java:2496)
    
    
    Looking later in the SystemOut.log, there will be more hung
    thread notifications which show that the
    FFDCFilter.processException() method is stuck waiting on an
    Ffdc.<init> call to complete.  The stack trace of such a thread
    will look like this:
    
    [05/24/12 12:42:35:914 EST] 00000003 ThreadMonitor W
    WSVR0605W: Thread "ORB.thread.pool : 29" (000007f4) has been
    active for 224151 milliseconds and may be hung.  There is/are 23
    thread(s) in total in the server that may be hung.
     at com.ibm.ffdc.util.provider.Ffdc.<init>(Ffdc.java:64)
     at
    com.ibm.ffdc.util.provider.FfdcProvider.getFfdc(FfdcProvider.jav
    a:289)
     at com.ibm.ffdc.impl.Ffdc.getFfdc(Ffdc.java:188)
     at
    com.ibm.ws.ffdc.FFDCFilter.processException(FFDCFilter.java:112)
     at
    com.ibm.ws.rsadapter.jdbc.WSJdbcPreparedStatement.pmiExecuteUpda
    te(WSJdbcPreparedStatement.java:1179)
     at
    com.ibm.ws.rsadapter.jdbc.WSJdbcPreparedStatement.executeUpdate(
    WSJdbcPreparedStatement.java:816)
    
    
    It is possible that the first FFDC thread which got hung (in
    this example, "ORB.thread.pool : 32") will eventually complete.
    If it does complete, the threads will start moving and the hang
    will clear up.  However, if it never completes, then it is the
    same as if a true thread deadlock had occurred.
    
    
    This same type of analysis can be performed using a
    javacore/thread dump, however, javacores are not always
    available.
    

Local fix

Problem summary

  • ****************************************************************
    * USERS AFFECTED:  All users of IBM WebSphere Application      *
    *                  Server V7.0 and V8.0                        *
    ****************************************************************
    * PROBLEM DESCRIPTION: Syncrhonization between FFDC            *
    *                      incidents can cause a server hang       *
    *                      when an unrecoverable problem           *
    *                      occurs in a given incident.             *
    ****************************************************************
    * RECOMMENDATION:                                              *
    ****************************************************************
    Syncrhonization between FFDC incidents can cause a server hang
    when an unrecoverable problem occurs in a given incident.
    

Problem conclusion

  • FFDC has been modified to ensure that incidents can continue
    to be logged from other components when a given component's
    incident-handling hangs a thread.  The impact to this is that
    the FFDC summary log may indicate that the incident file of a
    bad thread was created, while the thread may not have returned
    so the incident file will not exist.  Therefore, a missing
    incident entry that might have appeared in a summary file is
    an indication that the component logging to FFDC did not
    return control.
    
    The fix for this APAR is currently targeted for inclusion in
    fix packs 7.0.0.23 and 8.0.0.3.  Please refer to the
    Recommended Updates page for delivery information:
    http://www.ibm.com/support/docview.wss?rs=180&uid=swg27004980
    

Temporary fix

Comments

APAR Information

  • APAR number

    PM45011

  • Reported component name

    WEBS APP SERV N

  • Reported component ID

    5724H8800

  • Reported release

    700

  • Status

    CLOSED PER

  • PE

    NoPE

  • HIPER

    NoHIPER

  • Special Attention

    NoSpecatt

  • Submitted date

    2011-08-03

  • Closed date

    2011-12-15

  • Last modified date

    2012-06-01

  • APAR is sysrouted FROM one or more of the following:

  • APAR is sysrouted TO one or more of the following:

Fix information

  • Fixed component name

    WEBS APP SERV N

  • Fixed component ID

    5724H8800

Applicable component levels

  • R700 PSY

       UP

  • R800 PSY

       UP



Document information

More support for: WebSphere Application Server
General

Software version: 7.0

Reference #: PM45011

Modified date: 01 June 2012