IBM Support

PM60733: DEADLOCK HAPPENS WHEN TWO SEPERATE THREADS ATTEMPTING TO CLEAN UP THE MANAGED CONNECTIONS

Fixes are available

8.0.0.4: WebSphere Application Server V8.0 Fix Pack 4
7.0.0.25: WebSphere Application Server V7.0 Fix Pack 25
8.5.0.1: WebSphere Application Server V8.5 Fix Pack 1
8.0.0.5: WebSphere Application Server V8.0 Fix Pack 5
7.0.0.27: WebSphere Application Server V7.0 Fix Pack 27
8.5.0.2: WebSphere Application Server V8.5 Fix Pack 2
8.0.0.6: WebSphere Application Server V8.0 Fix Pack 6
7.0.0.29: WebSphere Application Server V7.0 Fix Pack 29
8.0.0.7: WebSphere Application Server V8.0 Fix Pack 7
8.0.0.8: WebSphere Application Server V8.0 Fix Pack 8
7.0.0.31: WebSphere Application Server V7.0 Fix Pack 31
7.0.0.27: Java SDK 1.6 SR13 FP2 Cumulative Fix for WebSphere Application Server
7.0.0.33: WebSphere Application Server V7.0 Fix Pack 33
8.0.0.9: WebSphere Application Server V8.0 Fix Pack 9
7.0.0.35: WebSphere Application Server V7.0 Fix Pack 35
8.0.0.10: WebSphere Application Server V8.0 Fix Pack 10
7.0.0.37: WebSphere Application Server V7.0 Fix Pack 37
8.0.0.11: WebSphere Application Server V8.0 Fix Pack 11
7.0.0.39: WebSphere Application Server V7.0 Fix Pack 39
8.0.0.12: WebSphere Application Server V8.0 Fix Pack 12
7.0.0.41: WebSphere Application Server V7.0 Fix Pack 41
8.0.0.13: WebSphere Application Server V8.0 Fix Pack 13
7.0.0.43: WebSphere Application Server V7.0 Fix Pack 43
8.0.0.14: WebSphere Application Server V8.0 Fix Pack 14
7.0.0.45: WebSphere Application Server V7.0 Fix Pack 45
8.0.0.15: WebSphere Application Server V8.0 Fix Pack 15
7.0.0.25: Java SDK 1.6 SR11 Cumulative Fix for WebSphere Application Server
7.0.0.27: Java SDK 1.6 SR12 Cumulative Fix for WebSphere Application Server
7.0.0.29: Java SDK 1.6 SR13 FP2 Cumulative Fix for WebSphere Application Server
7.0.0.45: Java SDK 1.6 SR16 FP60 Cumulative Fix for WebSphere Application Server
7.0.0.31: Java SDK 1.6 SR15 Cumulative Fix for WebSphere Application Server
7.0.0.35: Java SDK 1.6 SR16 FP1 Cumulative Fix for WebSphere Application Server
7.0.0.37: Java SDK 1.6 SR16 FP3 Cumulative Fix for WebSphere Application Server
7.0.0.39: Java SDK 1.6 SR16 FP7 Cumulative Fix for WebSphere Application Server
7.0.0.41: Java SDK 1.6 SR16 FP20 Cumulative Fix for WebSphere Application Server
7.0.0.43: Java SDK 1.6 SR16 FP41 Cumulative Fix for WebSphere Application Server

Subscribe

You can track all active APARs for this component.

APAR status

  • Closed as program error.

Error description

  • Deadlock was observed when two seperate threads attempting to
    cleanup the same managed MQ connection (JMSManagedSession):
    
    2LKDEADLOCKTHR  Thread "Thread-1285" (0x0000000072587D00)
    3LKDEADLOCKWTR    is waiting for:
    4LKDEADLOCKMON      sys_mon_t:0x000000012141B670 infl_mon_t:
    0x000000012141B6B0:
    4LKDEADLOCKOBJ
    java/util/Collections$SynchronizedSet@0x000000003438FC28/0x00000
    0003438FC34:
    3LKDEADLOCKOWN    which is owned by:
    2LKDEADLOCKTHR  Thread "MessageListenerThreadPool : 21"
    (0x0000000072A61400)
    3LKDEADLOCKWTR    which is waiting for:
    4LKDEADLOCKMON      sys_mon_t:0x000000012141B5E8 infl_mon_t:
    0x000000012141B628:
    4LKDEADLOCKOBJ
    com/ibm/ejs/jms/JMSQueueSessionHandle@0x000000003439E1A0/0x00000
    0003439E1AC:
    3LKDEADLOCKOWN    which is owned by:
    2LKDEADLOCKTHR  Thread "Thread-1285" (0x00000000725
    
    
    3XMTHREADINFO      "Thread-1285"
    J9VMThread:0x0000000072587D00, j9thread_t:0x0000000125243120,
    java/lang/Thread:0x00000000364F56E8, state:B, prio=5
    3XMTHREADINFO1            (native thread ID:0x3C1006F, native
    priority:0x5, native policy:UNKNOWN)
    3XMTHREADINFO3           Java callstack:
    4XESTACKTRACE                at
    com/ibm/ejs/jms/JMSManagedSession.cleanup(JMSManagedSession.java
    :522(Compiled Code))   <- waiting for openSessionHandle lock
    4XESTACKTRACE                at
    com/ibm/ejs/j2c/MCWrapper.cleanup(MCWrapper.java:1531(Compiled
    Code))
    4XESTACKTRACE                at
    com/ibm/ejs/j2c/FreePool.cleanupAndDestroyMCWrapper(FreePool.jav
    a:727)
    4XESTACKTRACE                at
    com/ibm/ejs/j2c/FreePool.returnToFreePool(FreePool.java:498(Comp
    iled Code))
    4XESTACKTRACE                at
    com/ibm/ejs/j2c/PoolManager.release(PoolManager.java:1853(Compil
    ed Code))
    4XESTACKTRACE                at
    com/ibm/ejs/j2c/MCWrapper.releaseToPoolManager(MCWrapper.java:23
    84(Compiled Code))
    4XESTACKTRACE                at
    com/ibm/ejs/j2c/MCWrapper.connectionErrorOccurred(MCWrapper.java
    :2524)
    4XESTACKTRACE                at
    com/ibm/ejs/j2c/ConnectionEventListener.connectionErrorOccurred(
    ConnectionEventListener.java:511)
    4XESTACKTRACE                at
    com/ibm/ejs/jms/JMSManagedSession.connectionErrorOccurred(JMSMan
    agedSession.java:1116)
    4XESTACKTRACE                at
    com/ibm/ejs/jms/JMSSessionHandle.markManagedObjectsAsStale(JMSSe
    ssionHandle.java:1520)     <-- this lock
    4XESTACKTRACE                at
    com/ibm/ejs/jms/JMSSessionHandle.closeChildren(JMSSessionHandle.
    java:1279(Compiled Code))
    4XESTACKTRACE                at
    com/ibm/ejs/jms/JMSSessionHandle.close(JMSSessionHandle.java:853
    )
    4XESTACKTRACE                at
    com/ibm/ejs/jms/JMSConnectionHandle.close(JMSConnectionHandle.ja
    va:561(Compiled Code))
    4XESTACKTRACE                at
    com/ibm/ejs/jms/listener/MDBListenerImpl.internalStop(MDBListene
    rImpl.java:995)
    4XESTACKTRACE                at
    com/ibm/ejs/jms/listener/JMSExceptionListener.onException(JMSExc
    eptionListener.java:127)
    4XESTACKTRACE                at
    com/ibm/ejs/jms/JMSManagedConnection.onException(JMSManagedConne
    ction.java:1391)
    4XESTACKTRACE                at
    com/ibm/msg/client/jms/internal/JmsProviderExceptionListener.run
    (JmsProviderExceptionListener.java:429)
    4XESTACKTRACE                at
    com/ibm/msg/client/commonservices/workqueue/WorkQueueItem.runTas
    k(WorkQueueItem.java:209)
    4XESTACKTRACE                at
    com/ibm/msg/client/commonservices/workqueue/SimpleWorkQueueItem.
    runItem(SimpleWorkQueueItem.java:100)
    4XESTACKTRACE                at
    com/ibm/msg/client/commonservices/workqueue/WorkQueueItem.run(Wo
    rkQueueItem.java:224)
    4XESTACKTRACE                at
    com/ibm/ws/wmqcsi/workqueue/WorkQueueManagerImpl$WorkQueueRunnab
    le.run(WorkQueueManagerImpl.java:648)
    4XESTACKTRACE                at
    java/lang/Thread.run(Thread.java:736(Compiled Code))
    
    3XMTHREADINFO      "MessageListenerThreadPool : 21"
    J9VMThread:0x0000000072A61400, j9thread_t:0x00000001212CFB20,
    java/lang/Thread:0x00000000343210B0, state:B, prio=5
    3XMTHREADINFO1            (native thread ID:0x57700DF, native
    priority:0x5, native policy:UNKNOWN)
    3XMTHREADINFO3           Java callstack:
    4XESTACKTRACE                at
    com/ibm/ejs/jms/JMSSessionHandle.deactivate(JMSSessionHandle.jav
    a:1470)     <-- waiting for this lock
    4XESTACKTRACE                at
    com/ibm/ejs/jms/JMSManagedSession.cleanup(JMSManagedSession.java
    :528(Compiled Code))   <-- openSessionHandle lock
    4XESTACKTRACE                at
    com/ibm/ejs/j2c/MCWrapper.cleanup(MCWrapper.java:1531(Compiled
    Code))
    4XESTACKTRACE                at
    com/ibm/ejs/j2c/FreePool.cleanupAndDestroyMCWrapper(FreePool.jav
    a:727)
    4XESTACKTRACE                at
    com/ibm/ejs/j2c/FreePool.returnToFreePool(FreePool.java:498(Comp
    iled Code))
    4XESTACKTRACE                at
    com/ibm/ejs/j2c/PoolManager.release(PoolManager.java:1853(Compil
    ed Code))
    4XESTACKTRACE                at
    com/ibm/ejs/j2c/MCWrapper.releaseToPoolManager(MCWrapper.java:23
    84(Compiled Code))
    4XESTACKTRACE                at
    com/ibm/ejs/j2c/XATransactionWrapper.afterCompletion(XATransacti
    onWrapper.java:335(Compiled Code))
    4XESTACKTRACE                at
    com/ibm/tx/jta/RegisteredSyncs.coreDistributeAfter(RegisteredSyn
    cs.java:357(Compiled Code))
    4XESTACKTRACE                at
    com/ibm/ws/tx/jta/RegisteredSyncs.distributeAfter(RegisteredSync
    s.java:317(Compiled Code))
    4XESTACKTRACE                at
    com/ibm/tx/jta/TransactionImpl.distributeAfter(TransactionImpl.j
    ava:2942(Compiled Code))
    4XESTACKTRACE                at
    com/ibm/tx/jta/TransactionImpl.postCompletion(TransactionImpl.ja
    va:2901(Compiled Code))
    4XESTACKTRACE                at
    com/ibm/tx/jta/TransactionImpl.internalRollback(TransactionImpl.
    java:1954)
    4XESTACKTRACE                at
    com/ibm/tx/jta/TransactionImpl.internalRollback(TransactionImpl.
    java:1914)
    4XESTACKTRACE                at
    com/ibm/tx/jta/TransactionImpl.rollback(TransactionImpl.java:131
    6)
    4XESTACKTRACE                at
    com/ibm/ws/tx/jta/TransactionImpl.rollback(TransactionImpl.java:
    3703)
    4XESTACKTRACE                at
    com/ibm/ws/tx/jta/TranManagerImpl.rollback(TranManagerImpl.java:
    476)
    4XESTACKTRACE                at
    com/ibm/tx/jta/TranManagerSet.rollback(TranManagerSet.java:186)
    4XESTACKTRACE                at
    com/ibm/ejs/csi/TranStrategy.rollback(TranStrategy.java:1146)
    4XESTACKTRACE                at
    com/ibm/ejs/csi/TranStrategy.postInvoke(TranStrategy.java:184(Co
    mpiled Code))
    4XESTACKTRACE                at
    com/ibm/ejs/csi/TransactionControlImpl.postInvoke(TransactionCon
    trolImpl.java:570(Compiled Code))
    4XESTACKTRACE                at
    com/ibm/ejs/container/EJSContainer.postInvoke(EJSContainer.java:
    4558(Compiled Code))
    4XESTACKTRACE                at
    com/ibm/ejs/container/MDBWrapper.onMessage(MDBWrapper.java:111(C
    ompiled Code))
    4XESTACKTRACE                at
    com/ibm/ejs/container/MDBWrapper.onMessage(MDBWrapper.java:136(C
    ompiled Code))
    4XESTACKTRACE                at
    com/ibm/ejs/jms/listener/ServerSession.run(ServerSession.java:54
    5(Compiled Code))
    4XESTACKTRACE                at
    com/ibm/ws/util/ThreadPool$Worker.run(ThreadPool.java:1604(Compi
    led Code))
    

Local fix

  • N/A
    

Problem summary

  • ****************************************************************
    * USERS AFFECTED:  All users of IBM WebSphere Application      *
    *                  Server using the WMQ JMS Provider           *
    ****************************************************************
    * PROBLEM DESCRIPTION: A deadlock occurs when a JMS Session    *
    *                      encounters a problem and is then        *
    *                      closed by 2 seperate threads.           *
    ****************************************************************
    * RECOMMENDATION:                                              *
    ****************************************************************
    During the rollback of a JMS Session, then managed connection
    is called to cleanup and be destroyed from the managed
    connection pool. At the same time the MQ JMS exception handler
    is driven (due to the same issue that caused the rollback),
    and the exception handler attempts to also close the same JMS
    Session.
    The result is a deadlock between the two threads.
    

Problem conclusion

  • This APAR fixes the problem and ensures that the correct
    locking strategy is used when closing JMS Sessions.
    
    The fix for this APAR is currently targeted for inclusion in
    fix pack 7.0.0.25 and 8.0.0.4.  Please refer to the Recommended
    Updates page for delivery information:
    http://www.ibm.com/support/docview.wss?rs=180&uid=swg27004980
    

Temporary fix

Comments

APAR Information

  • APAR number

    PM60733

  • Reported component name

    WEBS APP SERV N

  • Reported component ID

    5724H8800

  • Reported release

    700

  • Status

    CLOSED PER

  • PE

    NoPE

  • HIPER

    NoHIPER

  • Special Attention

    NoSpecatt

  • Submitted date

    2012-03-20

  • Closed date

    2012-05-24

  • Last modified date

    2012-05-24

  • APAR is sysrouted FROM one or more of the following:

  • APAR is sysrouted TO one or more of the following:

Fix information

  • Fixed component name

    WEBS APP SERV N

  • Fixed component ID

    5724H8800

Applicable component levels

  • R700 PSY

       UP

  • R800 PSY

       UP



Document information

More support for: WebSphere Application Server
General

Software version: 7.0

Reference #: PM60733

Modified date: 24 May 2012