IT18436: Deadlock seen in MQ Java on session.close() when async consume in use and poison message cannot be put to DLQ

APAR status

• Closed as program error.

Error description

• If an application using the IBM MQ classes for JMS configures a
  JMS MessageListener to receive messages asynchronously, a
  deadlock can occur between the MQ Java client and the queue
  manager.
  
  This has been observed when the following conditions occur:
  
    a) a poison message is being passed to the asynchronous
  consume callback function in the MQ classes for JMS from the
  queue manager,
  
    b) that message cannot be requeued to either the backout
  requeue queue or the dead-letter queue,
  
    c) at the same time, a separate application thread attempts to
  close the JMS MessageConsumer, Session or Connection (or for the
  JMS 2.0 API, a JMSConsumer or JMSContext) associated with the
  MessageListener.
  
  A Javacore (thread dump) of the application will show the two
  threads that are deadlocked.  The threads below are taken from
  an MQ classes for JMS application that connects to a queue
  manager using the BINDINGS transport mode:
  
  Thread 1 - Closing a JMS Session:
  ----------------------------------
  "main" J9VMThread:0x0000000000010500,
  j9thread_t:0x000000001004F540,
  java/lang/Thread:0x00000000A00372C0, state:R, prio=5
      (java/lang/Thread getId:0x1, isDaemon:false)
      (native thread ID:0x443, native priority:0x5, native
  policy:UNKNOWN, vmstate:R, vm thread flags:0x00000001)
      (native stack address range from:0x00000FFF8B610000,
  to:0x00000FFF8BE10000, size:0x800000)
     CPU usage total: 4.614949438 secs, current
  category="Application"
     Heap bytes allocated since last GC cycle=0 (0x0)
  
  Java callstack:
    at com/ibm/mq/jmqi/local/internal/base/Native.MQCTL
  	   (Native Method)
    at com/ibm/mq/jmqi/local/LocalMQ.MQCTL(
  	   LocalMQ.java:3125)
    at com/ibm/mq/jmqi/monitoring/JmqiInterceptAdapter.
         MQCTL(JmqiInterceptAdapter.java:333)
    at com/ibm/msg/client/wmq/internal/WMQConsumerOwnerShadow.
         controlAsyncService(WMQConsumerOwnerShadow.java:163)
    at com/ibm/msg/client/wmq/internal/WMQConsumerOwnerShadow.
         stop(WMQConsumerOwnerShadow.java:463)
    at com/ibm/msg/client/wmq/internal/WMQSession.stop
         (WMQSession.java:1881)
    at com/ibm/msg/client/jms/internal/JmsSessionImpl.stop
         (JmsSessionImpl.java:2537)
    at com/ibm/msg/client/jms/internal/JmsSessionImpl.close
         (JmsSessionImpl.java:423)
    at com/ibm/msg/client/jms/internal/JmsSessionImpl.close
         (JmsSessionImpl.java:342)
    at com/ibm/mq/jms/MQSession.close(MQSession.java:275)
  
  Thread 2 - AsyncConsume Thread:
  --------------------------------
  "Jmqi AsyncConsume Thread. tid: 40"
  J9VMThread:0x0000000000394B00, j9thread_t:0x000000001051E120,
  java/lang/Thread:0x00000000A1FA94D0, state:P, prio=5
      (java/lang/Thread getId:0x3C, isDaemon:true)
      (native thread ID:0x457, native priority:0x5, native
  policy:UNKNOWN, vmstate:P, vm thread flags:0x00020001)
      (native stack address range from:0x00000FFF74020000,
  to:0x00000FFF740A0000, size:0x80000)
     CPU usage total: 0.815008996 secs, current
  category="Application"
  
  Parked on:
  java/util/concurrent/locks/ReentrantLock$NonfairSync@0x00000000A
  1EB4668
  Owned by: "main" (J9VMThread:0x0000000000010500,
  java/lang/Thread:0x00000000A00372C0)
     Heap bytes allocated since last GC cycle=524640 (0x80160)
  
  Java callstack:
    at sun/misc/Unsafe.park(Native Method)
    at java/util/concurrent/locks/LockSupport.park
         (LockSupport.java:198)
    at java/util/concurrent/locks/AbstractQueuedSynchronizer.
  
  parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:846)
    at java/util/concurrent/locks/AbstractQueuedSynchronizer.
         acquireQueued(AbstractQueuedSynchronizer.java:879)
    at java/util/concurrent/locks/AbstractQueuedSynchronizer.
         acquire(AbstractQueuedSynchronizer.java:1209)
    at java/util/concurrent/locks/ReentrantLock$NonfairSync.
         lock(ReentrantLock.java:226)
    at java/util/concurrent/locks/ReentrantLock.lock
         (ReentrantLock.java:302)
    at com/ibm/msg/client/wmq/internal/WMQConsumerOwnerShadow.
         controlAsyncService(WMQConsumerOwnerShadow.java:145)
    at com/ibm/msg/client/wmq/internal/WMQConsumerOwnerShadow.
         removeAsyncConsumer(WMQConsumerOwnerShadow.java:305)
    at com/ibm/msg/client/wmq/internal/WMQSession.
         removeAsyncConsumer(WMQSession.java:1739)
    at com/ibm/msg/client/wmq/internal/WMQAsyncConsumerShadow.
         poisonous(WMQAsyncConsumerShadow.java:795)
    at com/ibm/msg/client/wmq/internal/WMQAsyncConsumerShadow.
         consumer(WMQAsyncConsumerShadow.java:501)
    at com/ibm/mq/jmqi/local/internal/LocalProxyConsumer.
         jmqiConsumerMethod(LocalProxyConsumer.java:207(Compiled
  Code))
  
  
  If this scenario occurs, the MQ classes for JMS log file
  (mqjms.log.0) will contain an error entry with following
  information:
  
  com.ibm.msg.client.jms.DetailedJMSException: JMSWMQ0032:
  There is no dead letter queue defined.
  
  WebSphere MQ classes for JMS attempted to put a poisonous
  message to the dead letter queue because it could not be routed
  to the backout retry queue. This failed because there was no
  dead letter queue defined to the queue manager.
  Define a dead letter queue.
  
  Class : class com.ibm.msg.client.jms.DetailedJMSException
  Stack :
  sun.reflect.NativeConstructorAccessorImpl.newInstance0
    (NativeConstructorAccessorImpl.java)
  sun.reflect.NativeConstructorAccessorImpl.newInstance
    (NativeConstructorAccessorImpl.java)
  sun.reflect.DelegatingConstructorAccessorImpl.newInstance
    (DelegatingConstructorAccessorImpl.java)
  java.lang.reflect.Constructor.newInstance
    (Constructor.java)
  com.ibm.msg.client.commonservices.j2se.NLSServices.
    createException(NLSServices.java)
  com.ibm.msg.client.commonservices.nls.NLSServices.
    createException(NLSServices.java:226)
  com.ibm.msg.client.wmq.internal.WMQPoison.deadletter
    (WMQPoison.java:834)
  com.ibm.msg.client.wmq.internal.WMQPoison.
    handlePoisonMessage(WMQPoison.java:483)
  com.ibm.msg.client.wmq.internal.WMQPoison.
    handlePoisonMessage(WMQPoison.java:264)
  com.ibm.msg.client.wmq.internal.WMQPoison.
    handlePoisonMessage(WMQPoison.java:237)
  com.ibm.msg.client.wmq.internal.WMQAsyncConsumerShadow.
    poisonous(WMQAsyncConsumerShadow.java:749)
  com.ibm.msg.client.wmq.internal.WMQAsyncConsumerShadow.
    consumer(WMQAsyncConsumerShadow.java:501)
  com.ibm.mq.jmqi.local.internal.LocalProxyConsumer.
    jmqiConsumerMethod(LocalProxyConsumer.java:207)
  
  EXPLANATION:
  The WebSphere MQ classes for JMS connection browser caught an
  exception.
  
  
  If the JMS ConnectionFactory used by the application set the
  ASYNC_EXCEPTIONS property to the value
  WMQConstants.ASYNC_EXCEPTIONS_ALL, the the JMS ExceptionListener
  for the application will be invoked with the following
  exception:
  
  com.ibm.msg.client.jms.DetailedJMSException: JMSWMQ0032: There
  is no dead letter queue defined.
    WebSphere MQ classes for JMS attempted to put a poisonous
  message to the dead letter queue because it could not be routed
  to the backout retry queue. This failed because there was no
  dead letter queue defined to the queue manager.
    Define a dead letter queue.
  

Local fix

• Ensure that poison messages can be requeued to either a backout
  requeue queue or the dead-letter queue.
  

Problem summary

• ****************************************************************
  USERS AFFECTED:
  This issue affects users of the IBM MQ classes for JMS who use
  JMS MessageListener's to receive messages asynchronously and are
  using a version of the MQ classes for JMS that have the fix for
  APAR IT10205 (e.g., IBM MQ V8.0.0.5, V9.0.0.0 or V9.0.1).
  
  
  Platforms affected:
  MultiPlatform
  
  ****************************************************************
  PROBLEM DESCRIPTION:
  After APAR IT10205, a new internal lock (MessageListener MQCTL
  lock) was added to the MQ classes for JMS to control the flow of
  MQCTL calls to a queue manager for JMS MessageListeners.  The
  new lock could have caused a deadlock if an application thread
  was closing either the MessageConsumer, JMSConsumer, Session,
  Connection or Context that was associated with the
  MessageListener at the same time as an asynchronous consume
  thread had been given a poison message to process that it could
  not move to the defined backout requeue queue or the queue
  manager's dead-letter queue (for example because it wasn't
  defined, was full or the application did not have permission).
  
  This was because the application thread stopping the JMS
  MessageListener would obtain the new MessageListener MQCTL lock
  and then make the MQCTL call as required.  The queue manager
  would not complete this MQCTL call until any outstanding
  messages have been acknowledged.  The asynchronous consume
  thread (the thread that delivers messages to a JMS
  MessageListener), after failing to requeue the outstanding
  poison message to either the backout requeue queue or the
  dead-letter queue, would then also attempt to stop any further
  message delivery by making its own MQCTL call.  The asynchronous
  consume thread, erroneously attempted to obtain the
  MessageListener MQCTL lock but blocked because it was held by an
  application thread.  The application thread, holding the
  MessageListener MQCTL lock couldn't complete until the
  asynchronous consume thread completes and so both threads became
  deadlocked.
  

Problem conclusion

• The MQ classes for JMS have been updated such that the
  asynchronous consume thread does not attempt to obtain the
  MessageListener MQCTL lock that was added as part of APAR
  IT10205.  This allows the asynchronous consume thread to
  complete its processing which, in turn, allows application
  threads closing JMS MessageListeners to also complete.
  
  ---------------------------------------------------------------
  The fix is targeted for delivery in the following PTFs:
  
  Version    Maintenance Level
  v8.0       8.0.0.6
  v9.0 CD    9.0.2
  v9.0 LTS   9.0.0.1
  
  The latest available maintenance can be obtained from
  'WebSphere MQ Recommended Fixes'
  http://www-1.ibm.com/support/docview.wss?rs=171&uid=swg27006037
  
  If the maintenance level is not yet available information on
  its planned availability can be found in 'WebSphere MQ
  Planned Maintenance Release Dates'
  http://www-1.ibm.com/support/docview.wss?rs=171&uid=swg27006309
  ---------------------------------------------------------------
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  WMQ BASE MULTIP

• Fixed component ID

  5724H7251

Applicable component levels

• R800 PSY

     UP