PM17856: DEADLOCK DETECTED BETWEEN WEBSPHERE APPLICATION SERVER V7 SERVICE INTEGRATION BUS THREADS

Fixes are available

APAR status

Closed as program error.

Error description

Websphere Application Server V7.0.0.x hosting a JMS application
reports hung threads.The following messages are written to the
SystemOut.log :
WSVR0605W: Thread "SIBJMSRAThreadPool : 8" (0000002b) has been
active for 688067 milliseconds and may be hung.  There is/are
X thread(s) in total in the server that may be hung.
WSVR0605W: Thread "Default : 0" (00000027) has been active for
687844 milliseconds and may be hung.  There is/are X thread(s)
in total in the server that may be hung.

Javacores show deadlock detected between the following threads:

3XMTHREADINFO      "Asynchronous Consumer : 0" J9VMThread:
0x00000000320D6700, j9thread_t:0x000000011CED0480, java/lang/
Thread:0x0000000043ED0290, state:B, prio=5
3XMTHREADINFO1            (native thread ID:0x5F204F, native
priority:0x5, native policy:UNKNOWN)
3XMTHREADINFO3           Java callstack:
4XESTACKTRACE                at com/ibm/ws/sib/ra/inbound/impl/
SibRaSingleProcessListener$SibRaWork.workAccepted(SibRaSingle
ProcessListener.java:621)
4XESTACKTRACE                at com/ibm/ejs/j2c/work/WorkProxy.
<init>(WorkProxy.java:184)
4XESTACKTRACE                at com/ibm/ejs/j2c/work/Work
ManagerImpl.scheduleWork(WorkManagerImpl.java:356)
4XESTACKTRACE                at com/ibm/ws/sib/ra/inbound/
impl/SibRaSingleProcessListener$SibRaWork.schedule(SibRa
SingleProcessListener.java:511)
4XESTACKTRACE                at com/ibm/ws/sib/ra/inbound/
impl/SibRaSingleProcessListener.internalConsumeMessages
(SibRaSingleProcessListener.java:327)

"SIBJMSRAThreadPool : 3" J9VMThread:0x000000003210A900,
j9thread_t:0x000000011DFD0A20, java/lang/Thread:0x0000000040
BAD550, state:B, prio=5
3XMTHREADINFO1            (native thread ID:0x2E40C5, native
priority:0x5, native policy:UNKNOWN)
3XMTHREADINFO3           Java callstack:
4XESTACKTRACE                at com/ibm/ws/sib/comms/client/
ConsumerSessionProxy.start(ConsumerSessionProxy.java:1187)
4XESTACKTRACE                at com/ibm/ws/sib/ra/inbound/
impl/SibRaSingleProcessListener$SibRaWork.workEnded(SibRa
SingleProcessListener.java:744)
4XESTACKTRACE                at com/ibm/ws/sib/ra/inbound/
impl/SibRaSingleProcessListener$SibRaWork.workCompleted
(SibRaSingleProcessListener.java:717)
4XESTACKTRACE                at com/ibm/ejs/j2c/work/Work
Proxy.run(WorkProxy.java:471)
4XESTACKTRACE                at com/ibm/ws/util/ThreadPool
$Worker.run(ThreadPool.java:1550)

Local fix

Problem summary

****************************************************************
* USERS AFFECTED:  Users of the default messaging provider for *
*                  IBM WebSphere Application Server Version    *
*                  7.0, with WebSphere MQ Server members of a  *
*                  Service Integration Bus                     *
****************************************************************
* PROBLEM DESCRIPTION: Message delivery stops unexpectedly     *
*                      when consuming from a WebSphere MQ      *
*                      Server destination                      *
****************************************************************
* RECOMMENDATION:  All customers with WebSphere MQ Server bus  *
*                  members should apply a fix for this APAR.   *
****************************************************************
The hang between SibRaWork.workEnded and
SibRaWork.workAccepted is just one possible symptom
of the issue. The hang occurs because of the following earlier
exception that prevented an attempt to suspend a consumer from
completing, and left the bus resource adapter servicing the
MDB in an inconsistent state:
om.ibm.wsspi.sib.core.exception.SIConnectionDroppedException:
CWSIJ0047E: An operation was attempted on a connection that is
already closed.
at com.ibm.ws.sib.jfapchannel.impl.ConversationImpl.
                       sendInternal(ConversationImpl.java:1121)
at com.ibm.ws.sib.jfapchannel.impl.ConversationImpl.
                            exchange(ConversationImpl.java:976)
at
com.ibm.ws.sib.comms.common.JFAPCommunicator.
                        jfapExchange(JFAPCommunicator.java:384)
at
com.ibm.ws.sib.comms.client.proxyqueue.impl.ConversationHelperIm
                pl.exchangeStop(ConversationHelperImpl.java:447)
at com.ibm.ws.sib.comms.client.proxyqueue.impl.
AsynchConsumerProxyQueueImpl.stopping(
                          AsynchConsumerProxyQueueImpl.java:674)
at com.ibm.ws.sib.comms.client.ConsumerSessionProxy.
                    stopInternal(ConsumerSessionProxy.java:1440)
at com.ibm.ws.sib.comms.client.ConsumerSessionProxy.
                            stop(ConsumerSessionProxy.java:1321)
at com.ibm.ws.sib.comms.client.ConsumerSessionProxy.
         performInCallbackActions(ConsumerSessionProxy.java:354)
at com.ibm.ws.sib.comms.client.proxyqueue.queue.
  AsynchConsumerQueue.deliverBatch(AsynchConsumerQueue.java:494)
at com.ibm.ws.sib.comms.client.proxyqueue.impl.
                  AsynchConsumerProxyQueueImpl.deliverMessages(
                         AsynchConsumerProxyQueueImpl.java:935)
...

The above exception is in turn caused by the following
exception that caused the network connection to be marked
broken, as more than one message was received over the network
from the messaging engine when only one message was expected:
com.ibm.websphere.sib.exception.SIErrorException: CWSIC1031E:
An internal error occurred.
at com.ibm.ws.sib.comms.client.proxyqueue.queue.
          AsynchConsumerQueue.put(AsynchConsumerQueue.java:139)
at com.ibm.ws.sib.comms.client.proxyqueue.impl.
 AsynchConsumerProxyQueueImpl.put(
                          AsynchConsumerProxyQueueImpl.java:456)
at com.ibm.ws.sib.comms.client.ProxyReceiveListener.
                   processMessage(ProxyReceiveListener.java:483)
at com.ibm.ws.sib.comms.client.ProxyReceiveListener.
                    dataReceived(ProxyReceiveListener.java:189)
...

The root cause is within the messaging engine, which is acting
as a proxy for the WebSphere MQ queue manager. It incorrectly
sends more than one message to the client, when only one
message is requested. The problem occurs because the code
performs a synchronous check for messages, before starting a
thread to poll for new messages arriving on the queue, but did
not check if the consumer had been suspended by the code
processing a message found during the sychronous check. This
caused additional messages to be send to the client, even
though it had suspended the consumer in the ME after the first
message was delivered.

Problem conclusion

The fix for this APAR corrects the WebSphere MQ Server bus
member code within the messaging engine, so that it does not
start the polling thread after a consumer has already been
suspended during a synchronous check of the queue.

The fix for this APAR is currently targeted for inclusion in
fix pack 7.0.0.13.  Please refer to the Recommended Updates
page for delivery information:
http://www.ibm.com/support/docview.wss?rs=180&uid=swg27004980

Temporary fix

Comments

APAR Information

APAR number
PM17856
Reported component name
PLAT MSG COM
Reported component ID
620800101
Reported release
300
Status
CLOSED PER
PE
NoPE
HIPER
NoHIPER
Special Attention
NoSpecatt
Submitted date
2010-07-06
Closed date
2010-08-31
Last modified date
2010-08-31

APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:

PM22969

Fix information

Fixed component name
PLAT MSG COM
Fixed component ID
620800101

Applicable component levels

R300 PSY
UP

[{"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Product":{"code":"SSEQTP","label":"WebSphere Application Server"},"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"7.0","Line of Business":{"code":"LOB45","label":"Automation"}}]

Document Information

Modified date:
24 October 2021

Tips

PM17856: DEADLOCK DETECTED BETWEEN WEBSPHERE APPLICATION SERVER V7 SERVICE INTEGRATION BUS THREADS

Fixes are available

Subscribe

APAR status

Closed as program error.

Error description

Local fix

Problem summary

Problem conclusion

Temporary fix

Comments

APAR Information

APAR number

Reported component name

Reported component ID

Reported release

Status

PE

HIPER

Special Attention

Submitted date

Closed date

Last modified date

APAR is sysrouted FROM one or more of the following:

APAR is sysrouted TO one or more of the following:

Fix information

Fixed component name

Fixed component ID

Applicable component levels

R300 PSY

Document Information

Share your feedback

Need support?