PM94352: DEADLOCK WHEN JMS SESSION IS CLOSED ON A SEPARATE THREAD FROM WHICH IT WAS OPENED.

APAR status

• Closed as program error.

Error description

• In WebSphere Application Server V7.0.0.25 - V7.0.0.29 (which
  include the fix for APAR PM60733), if a JMS connection is closed
  on a separate thread to which a Session derived from that
  Connection is being cleaned up (presumably because of some
  connection error), a JVM deadlock may occur.  This can result in
  messages not being received from a third-party JMS provider.
  
  Here is an example of two threads in a javacore. One catches a
  connection exception and the other tries to clean up the
  session/connection.
  
  Here is the deadlock notification:
  
  Deadlock detected !!!
  ---------------------
  Thread "MyTask" (0x0000000002664800)
  is waiting for:
  sys_mon_t:0x00002AD0622D3470 infl_mon_t: 0x00002AD0622D34E0:
  java/util/Collections$SynchronizedSet@0x0000000016D7ACB0/0x00000
  00016D7ACBC:
  which is owned by:
  Thread "MyOtherTask[1]" (0x0000000001A39300)
  which is waiting for:
  sys_mon_t:0x00002AD0622D33D0 infl_mon_t: 0x00002AD0622D3440:
  java/lang/Object@0x0000000016D7AD78/0x0000000016D7AD84:
  which is owned by:
  Thread "MyTask" (0x0000000002664800)
  ##
  
  Here is the thread that catches a connection exception:
  
  "MyTask" J9VMThread:0x0000000002664800,
  j9thread_t:0x00002AD052E38350,
  java/lang/Thread:0x0000000016D7A510, state:B, prio=5
  ...
  Java callstack:
  at
  com/ibm/ejs/jms/JMSManagedSession.connectionErrorOccurred(JMSMan
  agedSession.java:1141)
  at
  com/ibm/ejs/jms/JMSSessionHandle.markManagedObjectsAsStale(JMSSe
  ssionHandle.java:1535)
  at
  com/ibm/ejs/jms/JMSMessageConsumerHandle.markManagedObjectsAsSta
  le(JMSMessageConsumerHandle.java:915)
  at
  com/ibm/ejs/jms/JMSMessageConsumerHandle.receive(JMSMessageConsu
  merHandle.java:570(Compiled Code))
  at
  com/ibm/ejs/jms/JMSMessageConsumerHandle.receive(JMSMessageConsu
  merHandle.java:500(Compiled Code))
  at
  org/springframework/jms/listener/AbstractPollingMessageListenerC
  ontainer.receiveMessage(AbstractPollingMessageListenerContainer.
  java:405(Compiled Code))
  at
  org/springframework/jms/listener/AbstractPollingMessageListenerC
  ontainer.doReceiveAndExecute(AbstractPollingMessageListenerConta
  iner.java:308(Compiled Code))
  at
  org/springframework/jms/listener/AbstractPollingMessageListenerC
  ontainer.receiveAndExecute(AbstractPollingMessageListenerContain
  er.java:261(Compiled Code))
  at
  org/springframework/jms/listener/DefaultMessageListenerContainer
  $AsyncMessageListenerInvoker.invokeListener(DefaultMessageListen
  erContainer.java:982(Compiled Code))
  at
  org/springframework/jms/listener/DefaultMessageListenerContainer
  $AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageLi
  stenerContainer.java:974(Compiled Code))
  at
  org/springframework/jms/listener/DefaultMessageListenerContainer
  $AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer
  .java:876)
  at java/lang/Thread.run(Thread.java:738)
  ##
  
  Here is the thread that is doing the cleanup:
  
  "MyOtherTask[1]" J9VMThread:0x0000000001A39300,
  j9thread_t:0x00002AD0528F5760,
  java/lang/Thread:0x0000000016D7A830, state:B, prio=5
  ...
  Java callstack:
  at
  com/ibm/ejs/jms/JMSManagedSession.cleanup(JMSManagedSession.java
  :545(Compiled Code))
  at
  com/ibm/ejs/j2c/MCWrapper.cleanup(MCWrapper.java:1549(Compiled
  Code))
  at
  com/ibm/ejs/j2c/FreePool.returnToFreePool(FreePool.java:513(Comp
  iled Code))
  at
  com/ibm/ejs/j2c/PoolManager.release(PoolManager.java:1915(Compil
  ed Code))
  at
  com/ibm/ejs/j2c/MCWrapper.releaseToPoolManager(MCWrapper.java:24
  04(Compiled Code))
  at
  com/ibm/ejs/j2c/ConnectionEventListener.connectionClosed(Connect
  ionEventListener.java:377(Compiled Code))
  at
  com/ibm/ejs/jms/JMSManagedSession.handleClosed(JMSManagedSession
  .java:913(Compiled Code))
  at
  com/ibm/ejs/jms/JMSSessionHandle.close(JMSSessionHandle.java:871
  (Compiled Code))
  at
  com/ibm/ejs/jms/JMSConnectionHandle.close(JMSConnectionHandle.ja
  va:576(Compiled Code))
  at
  org/springframework/jms/connection/ConnectionFactoryUtils.releas
  eConnection(ConnectionFactoryUtils.java:81(Compiled Code))
  at
  org/springframework/jms/listener/AbstractJmsListeningContainer.r
  efreshSharedConnection(AbstractJmsListeningContainer.java:360)
  at
  org/springframework/jms/listener/DefaultMessageListenerContainer
  .refreshConnectionUntilSuccessful(DefaultMessageListenerContaine
  r.java:785)
  at
  org/springframework/jms/listener/DefaultMessageListenerContainer
  .recoverAfterListenerSetupFailure(DefaultMessageListenerContaine
  r.java:767)
  at
  org/springframework/jms/listener/DefaultMessageListenerContainer
  $AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer
  .java:898)
  at java/lang/Thread.run(Thread.java:738)
  

Local fix

Problem summary

• ****************************************************************
  * USERS AFFECTED:  All users of IBM WebSphere Application      *
  *                  Server JMS other than Service Integration   *
  *                  Bus JMS.                                    *
  ****************************************************************
  * PROBLEM DESCRIPTION: Deadlock possible after PM60733.        *
  ****************************************************************
  * RECOMMENDATION:                                              *
  ****************************************************************
  PM60733 introduced a new lock to break a different deadlock,
  but in turn introduced this new deadlock.
  

Problem conclusion

• This APAR resolves the problem by fixing the deadlock from
  PM60733 in a different way.  Instead of using a new lock,
  locks are not taken at the points causing the original
  deadlock.  Instead other non-locking changes are made to avoid
  the need to take any locks.
  
  The fix for this APAR is currently targeted for inclusion in
  fix pack 7.0.0.31, 8.0.0.8 and 8.5.5.1.  Please refer to the
  Recommended Updates page for delivery information:
  
  http://www.ibm.com/support/docview.wss?rs=180&uid=swg27004980
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

  PM92408

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  WEBSPHERE FOR Z

• Fixed component ID

  5655I3500

Applicable component levels

• R700 PSY

     UP

• R800 PSY

     UP

• R850 PSY

     UP