IBM Support

PM52527: THREADS HANG WHEN TRYING TO LOG AN FFDC AFTER CONNECTION POOL MAXES OUT.

Fixes are available

7.0.0.23: WebSphere Application Server V7.0 Fix Pack 23
8.0.0.4: WebSphere Application Server V8.0 Fix Pack 4
7.0.0.25: WebSphere Application Server V7.0 Fix Pack 25
8.0.0.5: WebSphere Application Server V8.0 Fix Pack 5
7.0.0.27: WebSphere Application Server V7.0 Fix Pack 27
8.0.0.6: WebSphere Application Server V8.0 Fix Pack 6
7.0.0.29: WebSphere Application Server V7.0 Fix Pack 29
8.0.0.7: WebSphere Application Server V8.0 Fix Pack 7
8.0.0.8: WebSphere Application Server V8.0 Fix Pack 8
7.0.0.31: WebSphere Application Server V7.0 Fix Pack 31
7.0.0.27: Java SDK 1.6 SR13 FP2 Cumulative Fix for WebSphere Application Server
7.0.0.33: WebSphere Application Server V7.0 Fix Pack 33
8.0.0.9: WebSphere Application Server V8.0 Fix Pack 9
7.0.0.35: WebSphere Application Server V7.0 Fix Pack 35
8.0.0.10: WebSphere Application Server V8.0 Fix Pack 10
7.0.0.37: WebSphere Application Server V7.0 Fix Pack 37
8.0.0.11: WebSphere Application Server V8.0 Fix Pack 11
7.0.0.39: WebSphere Application Server V7.0 Fix Pack 39
8.0.0.12: WebSphere Application Server V8.0 Fix Pack 12
7.0.0.41: WebSphere Application Server V7.0 Fix Pack 41
8.0.0.13: WebSphere Application Server V8.0 Fix Pack 13
7.0.0.43: WebSphere Application Server V7.0 Fix Pack 43
8.0.0.14: WebSphere Application Server V8.0 Fix Pack 14
7.0.0.45: WebSphere Application Server V7.0 Fix Pack 45
8.0.0.15: WebSphere Application Server V8.0 Fix Pack 15
7.0.0.23: Java SDK 1.6 SR10 FP1 Cumulative Fix for WebSphere
7.0.0.25: Java SDK 1.6 SR11 Cumulative Fix for WebSphere Application Server
7.0.0.27: Java SDK 1.6 SR12 Cumulative Fix for WebSphere Application Server
7.0.0.29: Java SDK 1.6 SR13 FP2 Cumulative Fix for WebSphere Application Server
7.0.0.45: Java SDK 1.6 SR16 FP60 Cumulative Fix for WebSphere Application Server
7.0.0.31: Java SDK 1.6 SR15 Cumulative Fix for WebSphere Application Server
7.0.0.35: Java SDK 1.6 SR16 FP1 Cumulative Fix for WebSphere Application Server
7.0.0.37: Java SDK 1.6 SR16 FP3 Cumulative Fix for WebSphere Application Server
7.0.0.39: Java SDK 1.6 SR16 FP7 Cumulative Fix for WebSphere Application Server
7.0.0.41: Java SDK 1.6 SR16 FP20 Cumulative Fix for WebSphere Application Server
7.0.0.43: Java SDK 1.6 SR16 FP41 Cumulative Fix for WebSphere Application Server

Subscribe

You can track all active APARs for this component.

APAR status

  • Closed as program error.

Error description

  • When using the Sybase jConnect JDBC driver with WebSphere
    Application Server, there is the potential for threads to hang
    if the connection pool maxes out.
    
    When the connection pool is full, the connection manager tries
    to log an FFDC of the error.  This results in a call made back
    to the Sybase JDBC driver to introspect itself.
    
    As a part of this, the Sybase driver invokes a SQL statement to
    complete the .isClosed() method.  In some circumstances, this
    call to Sybase can hang.
    
    
    The following thread stack illustrates the hang:
    
    "WebContainer : 1" J9VMThread:0x0000000030ECB900,
    j9thread_t:0x0000000116162460,
    java/lang/Thread:0x0000000041FFF528, state:CW, prio=5
           (native thread ID:0x19C0079, native priority:0x5, native
                policy:UNKNOWN)
          Java callstack:
              at java/lang/Object.wait(Native Method)
              at java/lang/Object.wait(Object.java:196(Compiled
    Code))
              at com/sybase/jdbc4/utils/SyncQueue.take(Bytecode
    PC:98)
              at com/sybase/jdbc4/utils/SyncObj.giveToMe(Bytecode
    PC:6)
              at com/sybase/jdbc4/timedio/InStreamMgr.a(Bytecode
    PC:84)
              at
    com/sybase/jdbc4/timedio/InStreamMgr.doRead(Bytecode PC:3)
              at
    com/sybase/jdbc4/tds/TdsProtocolContext.getChunk(Bytecode
    PC:143)
              at com/sybase/jdbc4/tds/PduInputFormatter.a(Bytecode
    PC:24)
              at
    com/sybase/jdbc4/tds/PduInputFormatter.read(Bytecode
    PC:12(Compiled Code))
              at com/sybase/jdbc4/tds/TdsInputStream.read(Bytecode
    PC:4(Compiled Code))
              at
    com/sybase/jdbc4/tds/TdsInputStream.readUnsignedByte(Bytecode
    PC:1)
              at com/sybase/jdbc4/tds/Tds.nextResult(Bytecode
    PC:172)
              at
    com/sybase/jdbc4/jdbc/ResultGetter.nextResult(Bytecode
    PC:26)
              at
    com/sybase/jdbc4/jdbc/SybStatement.nextResult(Bytecode
    PC:16)
              at
    com/sybase/jdbc4/jdbc/SybStatement.nextResult(Bytecode
    PC:5)
              at
    com/sybase/jdbc4/jdbc/SybStatement.queryLoop(Bytecode PC:8)
              at
    com/sybase/jdbc4/jdbc/SybCallableStatement.executeQuery(Bytecode
    PC:32)
              at
    com/sybase/jdbc4/jdbc/SybConnection.isClosed(Bytecode
    PC:62)
              at
    com/sybase/jdbc4/jdbc/SybConnectionProxy.isClosed(Bytecode
    PC:13)
              at
    com/ibm/ws/rsadapter/spi/WSRdbManagedConnectionImpl.introspectSe
    lf(WSRdbManagedConnectionImpl.java:1325)
              at sun/reflect/NativeMethodAccessorImpl.invoke0(Native
    Method)
              at
    sun/reflect/NativeMethodAccessorImpl.invoke(NativeMethodAccessor
    Impl.java:48(Compiled Code))
              at
    sun/reflect/DelegatingMethodAccessorImpl.invoke(DelegatingMethod
    AccessorImpl.java:25(Compiled Code))
              at
    java/lang/reflect/Method.invoke(Method.java:600(Compiled Code))
              at
    com/ibm/ws/ffdc/impl/Formatters$IntrospectSelfFormatter$1.run(Fo
    rmatters.java:99)
              at
    com/ibm/ws/ffdc/impl/Formatters$IntrospectSelfFormatter$1.run(Fo
    rmatters.java:96)
              at
    java/security/AccessController.doPrivileged(AccessController.jav
    a:251(Compiled Code))
              at
    com/ibm/ws/ffdc/impl/Formatters$IntrospectSelfFormatter.formatTo
    (Formatters.java:96)
              at
    com/ibm/ffdc/util/provider/IncidentStream$2.writeTo(IncidentStre
    am.java:194)
              at
    com/ibm/ffdc/util/provider/IncidentStream.write(IncidentStream.j
    ava:198)
              at
    com/ibm/ffdc/util/provider/IncidentStream.write(IncidentStream.j
    ava:129(Compiled Code))
              at
    com/ibm/ffdc/util/formatting/JavaUtilFormatter.formatTo(JavaUtil
    Formatter.java:54)
              at
    com/ibm/ffdc/util/formatting/JavaUtilFormatter.formatTo(JavaUtil
    Formatter.java:41)
              at
    com/ibm/ffdc/util/provider/IncidentStream$2.writeTo(IncidentStre
    am.java:194)
              at
    com/ibm/ffdc/util/provider/IncidentStream.write(IncidentStream.j
    ava:198)
              at
    com/ibm/ffdc/util/provider/IncidentStream.write(IncidentStream.j
    ava:129(Compiled Code))
              at
    com/ibm/ffdc/util/formatting/Introspector.formatTo(Introspector.
    java:109)
              at
    com/ibm/ffdc/util/provider/IncidentStream$2.writeTo(IncidentStre
    am.java:194)
              at
    com/ibm/ffdc/util/provider/IncidentStream.write(IncidentStream.j
    ava:198)
              at
    com/ibm/ffdc/util/provider/IncidentStream.write(IncidentStream.j
    ava:129)
              at
    com/ibm/ws/ffdc/impl/IncidentStream.introspectAndWriteLine(Incid
    entStream.java:126)
              at
    com/ibm/ejs/j2c/DiagnosticModuleForJ2C.ffdcDumpMaxConnectionsRea
    ched(DiagnosticModuleForJ2C.java:281)
    
    
    The above thread stack comes from a thread dump.
    
    The below messages might be seen in the SystemOut.log:
    
    [11/2/11 20:32:56:260 GMT+05:30] 0000002b FreePool E J2CA0045E:
    Connection not available while invoking method
    createOrWaitForConnection for resource myDatasource.
    
    [11/2/11 20:32:56:336 GMT+05:30] 0000002b DMAdapter I
    com.ibm.ws.ffdc.impl.DMAdapter getAnalysisEngine FFDC1009I:
    Analysis Engine using data base:
    /usr/IBM/WebSphere/AppServer/profiles/myProfile/properties/logb
    r/ffdc/adv/ffdcdb.xml
    
    [11/2/11 20:43:26:238 GMT+05:30] 00000003 ThreadMonitor W
    WSVR0605W:
    Thread "WebContainer : 1" (0000002b) has been active for 635301
    milliseconds and may be hung. There is/are 1 thread(s) in total
    in the
    server that may be hung.
    at java.lang.Object.wait(Native Method)
    at java.lang.Object.wait(Object.java:196)
    at com.sybase.jdbc3.utils.SyncQueue.take(Unknown Source)
    at com.sybase.jdbc3.utils.SyncObj.giveToMe(Unknown Source)
    at com.sybase.jdbc3.timedio.InStreamMgr.a(Unknown Source)
    at com.sybase.jdbc3.timedio.InStreamMgr.doRead(Unknown Source)
    at com.sybase.jdbc3.tds.TdsProtocolContext.getChunk(Unknown
    Source)
    at com.sybase.jdbc3.tds.PduInputFormatter.a(Unknown Source)
    
    
    IMPORTANT NOTE:  This APAR does not fix the problem which is
    causing the connection pool to max out (J2CA0045E).  This APAR
    prevents the thread hang which occurs when logging an FFDC of
    the J2CA0045E error.
    

Local fix

Problem summary

  • ****************************************************************
    * USERS AFFECTED:  All users of IBM WebSphere Application      *
    *                  Server V7.0 and V8.0                        *
    ****************************************************************
    * PROBLEM DESCRIPTION: Threads hang when traying to log an     *
    *                      FFDC for a ConnectionTimeOutException   *
    ****************************************************************
    * RECOMMENDATION:                                              *
    ****************************************************************
    When the WebSphere Application Server connection pool is full,
    requests for connections start to timeout and result in a
    ConnectionTimeOutException. We attempt to create an FFDC log of
    the ConnectionTimeoutExcption. During this process, the
    connections in the pool are introspected and certain
    information is extracted for creating the FFDC logs. This
    requires calling JDBC driver methods like
    connection.isClosed() and connection.getMetaData. It is
    possible that these calls can result in a hang if the
    application threads that was originally using the connection
    is running a long running operation on the connection.
    

Problem conclusion

  • The problem has been resolved by avoiding the calls to the
    driver during the FFDC operation and instead gathering that
    information from the Application Server runtime itself.
    
    The fix for this APAR is currently targeted for inclusion in
    fix packs 7.0.0.23 and 8.0.0.4.  Please refer to the
    Recommended Updates page for delivery information:
    http://www.ibm.com/support/docview.wss?rs=180&uid=swg27004980
    

Temporary fix

Comments

APAR Information

  • APAR number

    PM52527

  • Reported component name

    WEBSPHERE APP S

  • Reported component ID

    5724J0800

  • Reported release

    700

  • Status

    CLOSED PER

  • PE

    NoPE

  • HIPER

    NoHIPER

  • Special Attention

    NoSpecatt

  • Submitted date

    2011-11-17

  • Closed date

    2012-02-15

  • Last modified date

    2012-08-01

  • APAR is sysrouted FROM one or more of the following:

  • APAR is sysrouted TO one or more of the following:

Fix information

  • Fixed component name

    WEBSPHERE APP S

  • Fixed component ID

    5724J0800

Applicable component levels

  • R700 PSY

       UP

  • R800 PSY

       UP



Document information

More support for: WebSphere Application Server
General

Software version: 7.0

Reference #: PM52527

Modified date: 01 August 2012