PM88418: DEADLOCK ENCOUNTERED DURING SERVER STARTUP WHEN ATTEMPTING TO PROCESS IN-COMMIT TRANSACTIONS

Obtain the fix for this APAR.

APAR status

• Closed as program error.

Error description

• A deadlock in the Control Region causes the Servant to timeout
  waiting on a response from the Adjunct during transaction
  recovery of IN-COMMIT transactions during server startup.
  
  A javacore of the Servant Region shows a thread with the
  following stack:
  
  4XESTACKTRACE          at java/lang/Object.wait(Native Method)
  4XESTACKTRACE          at java/lang/Object.wait(Object.java:199)
  4XESTACKTRACE          at
  com/ibm/ws/sib/jfapchannel/impl/ExchangeReceiveListener.waitToCo
  mplete(ExchangeReceiveListener.java:235)
  4XESTACKTRACE          at
  com/ibm/ws/sib/jfapchannel/impl/ConversationImpl.exchange(Conver
  sationImpl.java:918)
  4XESTACKTRACE          at
  com/ibm/ws/sib/comms/common/JFAPCommunicator.JFAPExchange(JFAPCo
  mmunicator.java:382)
  4XESTACKTRACE          at
  com/ibm/ws/sib/comms/client/BaseSIXAResourceProxy.internalRecove
  r(BaseSIXAResourceProxy.java:518)
  4XESTACKTRACE          at
  com/ibm/ws/sib/comms/client/OptimizedSIXAResourceProxy.recover(O
  ptimizedSIXAResourceProxy.java:883)
  4XESTACKTRACE          at
  com/ibm/ws/sib/ra/recovery/impl/SibRaRecoveryXaResource.recover(
  SibRaRecoveryXaResource.java:223)
  4XESTACKTRACE          at
  com/ibm/ws/Transaction/JTA/XARminst.recover(XARminst.java:138)
  4XESTACKTRACE          at
  com/ibm/ws390/tx/XARecoveryAgentImpl.commit(XARecoveryAgentImpl.
  java:965)
  4XESTACKTRACE          at
  com/ibm/ws390/tx/XARecoveryAgentImpl$XARecoveryAgentThread.run(X
  ARecoveryAgentImpl.java:358)
  
  The resource that is being committed is an MQ Server resource.
  

Local fix

Problem summary

• ****************************************************************
  * USERS AFFECTED:  All users of IBM WebSphere Application      *
  *                  Server V7.0 MQ Server Bus Members           *
  ****************************************************************
  * PROBLEM DESCRIPTION: Deadlock in Control Region leading to   *
  *                      ABEND when attempting to commit MQ      *
  *                      Server IN-COMMIT transaction.           *
  ****************************************************************
  * RECOMMENDATION:                                              *
  ****************************************************************
  MQ Server uses JCA 1.5 transaction in-flow interfaces as part
  of its normal operation.  These interfaces enable MQ Server
  Bus members to coordinate WMQ and Service Integration Bus
  resources using WebSphere Application Server transactions.
  During commit of a transaction at recovery time, xa_recover is
  called to ensure some older DB2 drivers do not return
  XAER_NOTA even if the the transaction does exist when
  xa_commit or xa_rollback is issued.
  The deadlock ensues after the xa_recover call is issued against
  an MQ Server resource at recovery time.  This is because the MQ
  Server resource calls back into the Control Region and
  attempts to lock all known transactions to generate a list of
  transactions needed to respond to the xa_recover call.  The
  IN-COMMIT transaction being committed is already locked so the
  xa_recover call hangs waiting to lock the IN-COMMIT
  transaction which will not be unlocked until the transaction
  commits.
  

Problem conclusion

• This APAR introduces a new custom property of the transaction
  service:
  
  ZOS_RECOVER_BEFORE_COMMIT (default true)
  
  The default value "true" has no effect.  When set to "false"
  this property prevents the xa_recover call prior to xa_commit
  or xa_rollback during the processing of transactions at
  recovery time.  This avoids the deadlock described by this APAR.
  
  To set this property:
  
  1. Login to the Administrative Console.
  2. Select Servers -> Application Servers -> <server> ->
     Container Services -> Transaction Service.
  3. Select the Configuration tab.
  4. Select Custom Properties.
  5. Click new.
  6. Enter ZOS_RECOVER_BEFORE_COMMIT as the name, and "false" as
     the value.
  7. Click Ok.
  8. Save changes.
  9. Restart the server.
  
  All up to date DB2 drivers have an auto-recover feature which
  makes xa_recover prior to xa_commit or xa_rollback
  unnecessary.  If unsure, or it is not clear from the
  documentation for the version of the DB2 driver used, consult
  IBM DB2 support for final clarification on whether setting
  this property is safe for servers hosting applications that
  access DB2.
  
  APAR PM88418 requires changes to documentation.
  
  NOTE: Periodically, we refresh the documentation on our Web
  site, so the changes might have been made before you read this
  text. To access the latest on-line documentation, go to the
  product library page at:
  
  http://www.ibm.com/software/webservers/appserv/library
  
  The following changes to the WebSphere Application Server
  Version 6.1 Information Center will be made available in
  June 2013.
  
  The topic "Transaction service custom properties" will be
  updated to include the following description of the
  ZOS_RECOVER_BEFORE_COMMIT custom property:
  
  ZOS_RECOVER_BEFORE_COMMIT
  
  Specifying this property prevents a deadlock from occurring
  after an xa_recover call is issued against an MQ Server
  resource at recovery time. The MQ Server uses JCA 1.5
  transaction in-flow interfaces as part of its normal
  operation. These interfaces enable MQ Server Bus members to
  coordinate WMQ and Service Integration Bus resources using
  WebSphere Application Server transactions. During the commit
  of any transaction at recovery time, xa_recover is called to
  ensure some older DB2 drivers do not return XAER_NOTA even if
  the transaction exists when the xa_commit or xa_rollback call
  is issued.
  
  This deadlock occurs because the MQ Server resource issues
  calls back to the controller and attempts to lock all known
  transactions so that the MQ Server resource can generate a
  list of transactions that need to respond to the xa_recover
  call. However, because the IN-COMMIT transaction being
  committed is already locked, the xa_recover call waits
  indefinitely to lock the IN-COMMIT transaction because that
  transaction will not be unlocked until the transaction commits.
  
  Setting this property to FALSE ensures that during the
  processing of transactions at recovery time, the xa_recover
  call is not issued before an xa_commit or xa_rollback call.
  Avoid Trouble: All currently supported DB2 drivers have an
  auto-recover feature which makes it unnecessary to issue an
  xa_recover calls prior to issuing an xa_commit or xa_rollback
  call. If, after reading the documentation for your DB2 driver,
  you are not sure whether your DB2 driver includes the
  auto-recover feature, contact IBM DB2 support for final
  clarification on whether setting this property is safe to use
  for servers hosting applications that access DB2.
  Table 8: ZOS_RECOVER_BEFORE_COMMIT custom property
  
  Data Type                    Boolean
  Accepted Values          TRUE, FALSE
  Default                          TRUE
  
  APAR PM88418 is currently targeted for inclusion in Fix Pack
  7.0.0.29 of WebSphere Application Server V7.0.
  
  Please refer to URL:
  //www.ibm.com/support/docview.wss?rs=404&uid=swg27006970
  for Fix Pack availability.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

  PM75705

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  WEBSPHERE FOR Z

• Fixed component ID

  5655I3500

Applicable component levels

• R700 PSY UK94926

     UP13/06/20 P F306

Fix is available

• Select the PTF appropriate for your component level. You will be required to sign in. Distribution on physical media is not available in all countries.