IBM Support

PM17856: DEADLOCK DETECTED BETWEEN WEBSPHERE APPLICATION SERVER V7 SERVICE INTEGRATION BUS THREADS

Fixes are available

7.0.0.13: WebSphere Application Server V7.0 Fix Pack 13 for AIX
7.0.0.13: WebSphere Application Server V7.0 Fix Pack 13 for HP-UX
7.0.0.13: WebSphere Application Server V7.0 Fix Pack 13 for IBM i
7.0.0.13: WebSphere Application Server V7.0 Fix Pack 13 for Linux
7.0.0.13: WebSphere Application Server V7.0 Fix Pack 13 for Solaris
7.0.0.13: WebSphere Application Server V7.0 Fix Pack 13 for Windows
7.0.0.13: Java SDK 1.6 SR8FP1 Cumulative Fix for WebSphere Application Server
7.0.0.15: WebSphere Application Server V7.0 Fix Pack 15 for AIX
7.0.0.15: Java SDK 1.6 SR9 Cumulative Fix for WebSphere Application Server
7.0.0.15: WebSphere Application Server V7.0 Fix Pack 15 for HP-UX
7.0.0.15: WebSphere Application Server V7.0 Fix Pack 15 for IBM i
7.0.0.15: WebSphere Application Server V7.0 Fix Pack 15 for Linux
7.0.0.15: WebSphere Application Server V7.0 Fix Pack 15 for Solaris
7.0.0.15: WebSphere Application Server V7.0 Fix Pack 15 for Windows
7.0.0.17: WebSphere Application Server V7.0 Fix Pack 17
7.0.0.17: Java SDK 1.6 SR9 FP1 Cumulative Fix for WebSphere Application Server
7.0.0.19: WebSphere Application Server V7.0 Fix Pack 19
7.0.0.21: WebSphere Application Server V7.0 Fix Pack 21
7.0.0.23: WebSphere Application Server V7.0 Fix Pack 23
7.0.0.25: WebSphere Application Server V7.0 Fix Pack 25
7.0.0.27: WebSphere Application Server V7.0 Fix Pack 27
7.0.0.29: WebSphere Application Server V7.0 Fix Pack 29
7.0.0.31: WebSphere Application Server V7.0 Fix Pack 31
7.0.0.27: Java SDK 1.6 SR13 FP2 Cumulative Fix for WebSphere Application Server
7.0.0.33: WebSphere Application Server V7.0 Fix Pack 33
7.0.0.35: WebSphere Application Server V7.0 Fix Pack 35
7.0.0.37: WebSphere Application Server V7.0 Fix Pack 37
7.0.0.39: WebSphere Application Server V7.0 Fix Pack 39
7.0.0.41: WebSphere Application Server V7.0 Fix Pack 41
7.0.0.43: WebSphere Application Server V7.0 Fix Pack 43
7.0.0.45: WebSphere Application Server V7.0 Fix Pack 45
7.0.0.19: Java SDK 1.6 SR9 FP2 Cumulative Fix for WebSphere Application Server
7.0.0.21: Java SDK 1.6 SR9 FP2 Cumulative Fix for WebSphere
7.0.0.23: Java SDK 1.6 SR10 FP1 Cumulative Fix for WebSphere
7.0.0.25: Java SDK 1.6 SR11 Cumulative Fix for WebSphere Application Server
7.0.0.27: Java SDK 1.6 SR12 Cumulative Fix for WebSphere Application Server
7.0.0.29: Java SDK 1.6 SR13 FP2 Cumulative Fix for WebSphere Application Server
7.0.0.45: Java SDK 1.6 SR16 FP60 Cumulative Fix for WebSphere Application Server
7.0.0.31: Java SDK 1.6 SR15 Cumulative Fix for WebSphere Application Server
7.0.0.35: Java SDK 1.6 SR16 FP1 Cumulative Fix for WebSphere Application Server
7.0.0.37: Java SDK 1.6 SR16 FP3 Cumulative Fix for WebSphere Application Server
7.0.0.39: Java SDK 1.6 SR16 FP7 Cumulative Fix for WebSphere Application Server
7.0.0.41: Java SDK 1.6 SR16 FP20 Cumulative Fix for WebSphere Application Server
7.0.0.43: Java SDK 1.6 SR16 FP41 Cumulative Fix for WebSphere Application Server

Subscribe

You can track all active APARs for this component.

APAR status

  • Closed as program error.

Error description

  • Websphere Application Server V7.0.0.x hosting a JMS application
    reports hung threads.The following messages are written to the
    SystemOut.log :
    WSVR0605W: Thread "SIBJMSRAThreadPool : 8" (0000002b) has been
    active for 688067 milliseconds and may be hung.  There is/are
    X thread(s) in total in the server that may be hung.
    WSVR0605W: Thread "Default : 0" (00000027) has been active for
    687844 milliseconds and may be hung.  There is/are X thread(s)
    in total in the server that may be hung.
    
    Javacores show deadlock detected between the following threads:
    
    3XMTHREADINFO      "Asynchronous Consumer : 0" J9VMThread:
    0x00000000320D6700, j9thread_t:0x000000011CED0480, java/lang/
    Thread:0x0000000043ED0290, state:B, prio=5
    3XMTHREADINFO1            (native thread ID:0x5F204F, native
    priority:0x5, native policy:UNKNOWN)
    3XMTHREADINFO3           Java callstack:
    4XESTACKTRACE                at com/ibm/ws/sib/ra/inbound/impl/
    SibRaSingleProcessListener$SibRaWork.workAccepted(SibRaSingle
    ProcessListener.java:621)
    4XESTACKTRACE                at com/ibm/ejs/j2c/work/WorkProxy.
    <init>(WorkProxy.java:184)
    4XESTACKTRACE                at com/ibm/ejs/j2c/work/Work
    ManagerImpl.scheduleWork(WorkManagerImpl.java:356)
    4XESTACKTRACE                at com/ibm/ws/sib/ra/inbound/
    impl/SibRaSingleProcessListener$SibRaWork.schedule(SibRa
    SingleProcessListener.java:511)
    4XESTACKTRACE                at com/ibm/ws/sib/ra/inbound/
    impl/SibRaSingleProcessListener.internalConsumeMessages
    (SibRaSingleProcessListener.java:327)
    
    "SIBJMSRAThreadPool : 3" J9VMThread:0x000000003210A900,
    j9thread_t:0x000000011DFD0A20, java/lang/Thread:0x0000000040
    BAD550, state:B, prio=5
    3XMTHREADINFO1            (native thread ID:0x2E40C5, native
    priority:0x5, native policy:UNKNOWN)
    3XMTHREADINFO3           Java callstack:
    4XESTACKTRACE                at com/ibm/ws/sib/comms/client/
    ConsumerSessionProxy.start(ConsumerSessionProxy.java:1187)
    4XESTACKTRACE                at com/ibm/ws/sib/ra/inbound/
    impl/SibRaSingleProcessListener$SibRaWork.workEnded(SibRa
    SingleProcessListener.java:744)
    4XESTACKTRACE                at com/ibm/ws/sib/ra/inbound/
    impl/SibRaSingleProcessListener$SibRaWork.workCompleted
    (SibRaSingleProcessListener.java:717)
    4XESTACKTRACE                at com/ibm/ejs/j2c/work/Work
    Proxy.run(WorkProxy.java:471)
    4XESTACKTRACE                at com/ibm/ws/util/ThreadPool
    $Worker.run(ThreadPool.java:1550)
    

Local fix

Problem summary

  • ****************************************************************
    * USERS AFFECTED:  Users of the default messaging provider for *
    *                  IBM WebSphere Application Server Version    *
    *                  7.0, with WebSphere MQ Server members of a  *
    *                  Service Integration Bus                     *
    ****************************************************************
    * PROBLEM DESCRIPTION: Message delivery stops unexpectedly     *
    *                      when consuming from a WebSphere MQ      *
    *                      Server destination                      *
    ****************************************************************
    * RECOMMENDATION:  All customers with WebSphere MQ Server bus  *
    *                  members should apply a fix for this APAR.   *
    ****************************************************************
    The hang between SibRaWork.workEnded and
    SibRaWork.workAccepted is just one possible symptom
    of the issue. The hang occurs because of the following earlier
    exception that prevented an attempt to suspend a consumer from
    completing, and left the bus resource adapter servicing the
    MDB in an inconsistent state:
    om.ibm.wsspi.sib.core.exception.SIConnectionDroppedException:
    CWSIJ0047E: An operation was attempted on a connection that is
    already closed.
    at com.ibm.ws.sib.jfapchannel.impl.ConversationImpl.
                           sendInternal(ConversationImpl.java:1121)
    at com.ibm.ws.sib.jfapchannel.impl.ConversationImpl.
                                exchange(ConversationImpl.java:976)
    at
    com.ibm.ws.sib.comms.common.JFAPCommunicator.
                            jfapExchange(JFAPCommunicator.java:384)
    at
    com.ibm.ws.sib.comms.client.proxyqueue.impl.ConversationHelperIm
                    pl.exchangeStop(ConversationHelperImpl.java:447)
    at com.ibm.ws.sib.comms.client.proxyqueue.impl.
    AsynchConsumerProxyQueueImpl.stopping(
                              AsynchConsumerProxyQueueImpl.java:674)
    at com.ibm.ws.sib.comms.client.ConsumerSessionProxy.
                        stopInternal(ConsumerSessionProxy.java:1440)
    at com.ibm.ws.sib.comms.client.ConsumerSessionProxy.
                                stop(ConsumerSessionProxy.java:1321)
    at com.ibm.ws.sib.comms.client.ConsumerSessionProxy.
             performInCallbackActions(ConsumerSessionProxy.java:354)
    at com.ibm.ws.sib.comms.client.proxyqueue.queue.
      AsynchConsumerQueue.deliverBatch(AsynchConsumerQueue.java:494)
    at com.ibm.ws.sib.comms.client.proxyqueue.impl.
                      AsynchConsumerProxyQueueImpl.deliverMessages(
                             AsynchConsumerProxyQueueImpl.java:935)
    ...
    
    The above exception is in turn caused by the following
    exception that caused the network connection to be marked
    broken, as more than one message was received over the network
    from the messaging engine when only one message was expected:
    com.ibm.websphere.sib.exception.SIErrorException: CWSIC1031E:
    An internal error occurred.
    at com.ibm.ws.sib.comms.client.proxyqueue.queue.
              AsynchConsumerQueue.put(AsynchConsumerQueue.java:139)
    at com.ibm.ws.sib.comms.client.proxyqueue.impl.
     AsynchConsumerProxyQueueImpl.put(
                              AsynchConsumerProxyQueueImpl.java:456)
    at com.ibm.ws.sib.comms.client.ProxyReceiveListener.
                       processMessage(ProxyReceiveListener.java:483)
    at com.ibm.ws.sib.comms.client.ProxyReceiveListener.
                        dataReceived(ProxyReceiveListener.java:189)
    ...
    
    The root cause is within the messaging engine, which is acting
    as a proxy for the WebSphere MQ queue manager. It incorrectly
    sends more than one message to the client, when only one
    message is requested. The problem occurs because the code
    performs a synchronous check for messages, before starting a
    thread to poll for new messages arriving on the queue, but did
    not check if the consumer had been suspended by the code
    processing a message found during the sychronous check. This
    caused additional messages to be send to the client, even
    though it had suspended the consumer in the ME after the first
    message was delivered.
    

Problem conclusion

  • The fix for this APAR corrects the WebSphere MQ Server bus
    member code within the messaging engine, so that it does not
    start the polling thread after a consumer has already been
    suspended during a synchronous check of the queue.
    
    The fix for this APAR is currently targeted for inclusion in
    fix pack 7.0.0.13.  Please refer to the Recommended Updates
    page for delivery information:
    http://www.ibm.com/support/docview.wss?rs=180&uid=swg27004980
    

Temporary fix

Comments

APAR Information

  • APAR number

    PM17856

  • Reported component name

    PLAT MSG COM

  • Reported component ID

    620800101

  • Reported release

    300

  • Status

    CLOSED PER

  • PE

    NoPE

  • HIPER

    NoHIPER

  • Special Attention

    NoSpecatt

  • Submitted date

    2010-07-06

  • Closed date

    2010-08-31

  • Last modified date

    2010-08-31

  • APAR is sysrouted FROM one or more of the following:

  • APAR is sysrouted TO one or more of the following:

    PM22969

Fix information

  • Fixed component name

    PLAT MSG COM

  • Fixed component ID

    620800101

Applicable component levels

  • R300 PSY

       UP



Document information

More support for: WebSphere Application Server
Service Integration Technology

Software version: 7.0

Reference #: PM17856

Modified date: 31 August 2010