IBM Support

PM75705: DEADLOCK ENCOUNTERED DURING SERVER STARTUP WHEN ATTEMPTING TO PROCESS IN-COMMIT TRANSACTIONS

Fixes are available

8.5.0.2: WebSphere Application Server V8.5 Fix Pack 2
8.0.0.6: WebSphere Application Server V8.0 Fix Pack 6
7.0.0.29: WebSphere Application Server V7.0 Fix Pack 29
8.0.0.7: WebSphere Application Server V8.0 Fix Pack 7
6.1.0.47: WebSphere Application Server V6.1 Fix Pack 47
8.0.0.8: WebSphere Application Server V8.0 Fix Pack 8
7.0.0.31: WebSphere Application Server V7.0 Fix Pack 31
7.0.0.33: WebSphere Application Server V7.0 Fix Pack 33
8.0.0.9: WebSphere Application Server V8.0 Fix Pack 9
7.0.0.35: WebSphere Application Server V7.0 Fix Pack 35
8.0.0.10: WebSphere Application Server V8.0 Fix Pack 10
7.0.0.37: WebSphere Application Server V7.0 Fix Pack 37
8.0.0.11: WebSphere Application Server V8.0 Fix Pack 11
7.0.0.39: WebSphere Application Server V7.0 Fix Pack 39
8.0.0.12: WebSphere Application Server V8.0 Fix Pack 12
7.0.0.41: WebSphere Application Server V7.0 Fix Pack 41
8.0.0.13: WebSphere Application Server V8.0 Fix Pack 13
7.0.0.43: WebSphere Application Server V7.0 Fix Pack 43
8.0.0.14: WebSphere Application Server V8.0 Fix Pack 14
7.0.0.45: WebSphere Application Server V7.0 Fix Pack 45
8.0.0.15: WebSphere Application Server V8.0 Fix Pack 15
6.1.0.47: Java SDK 1.5 SR16 Cumulative Fix for WebSphere Application Server
7.0.0.29: Java SDK 1.6 SR13 FP2 Cumulative Fix for WebSphere Application Server
7.0.0.45: Java SDK 1.6 SR16 FP60 Cumulative Fix for WebSphere Application Server
7.0.0.31: Java SDK 1.6 SR15 Cumulative Fix for WebSphere Application Server
7.0.0.35: Java SDK 1.6 SR16 FP1 Cumulative Fix for WebSphere Application Server
7.0.0.37: Java SDK 1.6 SR16 FP3 Cumulative Fix for WebSphere Application Server
7.0.0.39: Java SDK 1.6 SR16 FP7 Cumulative Fix for WebSphere Application Server
7.0.0.41: Java SDK 1.6 SR16 FP20 Cumulative Fix for WebSphere Application Server
7.0.0.43: Java SDK 1.6 SR16 FP41 Cumulative Fix for WebSphere Application Server
Obtain the fix for this APAR.

Subscribe

You can track all active APARs for this component.

APAR status

  • Closed as program error.

Error description

  • A deadlock in the Control Region causes the Servant to timeout
    waiting on a response from the Adjunct during transaction
    recovery of IN-COMMIT transactions during server startup.
    
    A javacore of the Servant Region shows a thread with the
    following stack:
    
    4XESTACKTRACE          at java/lang/Object.wait(Native Method)
    4XESTACKTRACE          at java/lang/Object.wait(Object.java:199)
    4XESTACKTRACE          at
    com/ibm/ws/sib/jfapchannel/impl/ExchangeReceiveListener.waitToCo
    mplete(ExchangeReceiveListener.java:235)
    4XESTACKTRACE          at
    com/ibm/ws/sib/jfapchannel/impl/ConversationImpl.exchange(Conver
    sationImpl.java:918)
    4XESTACKTRACE          at
    com/ibm/ws/sib/comms/common/JFAPCommunicator.JFAPExchange(JFAPCo
    mmunicator.java:382)
    4XESTACKTRACE          at
    com/ibm/ws/sib/comms/client/BaseSIXAResourceProxy.internalRecove
    r(BaseSIXAResourceProxy.java:518)
    4XESTACKTRACE          at
    com/ibm/ws/sib/comms/client/OptimizedSIXAResourceProxy.recover(O
    ptimizedSIXAResourceProxy.java:883)
    4XESTACKTRACE          at
    com/ibm/ws/sib/ra/recovery/impl/SibRaRecoveryXaResource.recover(
    SibRaRecoveryXaResource.java:223)
    4XESTACKTRACE          at
    com/ibm/ws/Transaction/JTA/XARminst.recover(XARminst.java:138)
    4XESTACKTRACE          at
    com/ibm/ws390/tx/XARecoveryAgentImpl.commit(XARecoveryAgentImpl.
    java:965)
    4XESTACKTRACE          at
    com/ibm/ws390/tx/XARecoveryAgentImpl$XARecoveryAgentThread.run(X
    ARecoveryAgentImpl.java:358)
    
    The resource that is being committed is an MQ Server resource.
    

Local fix

Problem summary

  • ****************************************************************
    * USERS AFFECTED:  All users of IBM WebSphere Application      *
    *                  Server V6.1, V8.0, and V8.5 MQ Server       *
    *                  Bus Members                                 *
    ****************************************************************
    * PROBLEM DESCRIPTION: Deadlock in Control Region leading to   *
    *                      ABEND when attempting to commit MQ      *
    *                      Server IN-COMMIT transaction.           *
    ****************************************************************
    * RECOMMENDATION:                                              *
    ****************************************************************
    MQ Server uses JCA 1.5 transaction in-flow interfaces as part
    of its normal operation.  These interfaces enable MQ Server
    Bus members to coordinate WMQ and Service Integration Bus
    resources using WebSphere Application Server transactions.
    During commit of a transaction at recovery time, xa_recover is
    called to ensure some older DB2 drivers do not return
    XAER_NOTA even if the the transaction does exist when
    xa_commit or xa_rollback is issued.
    
    The deadlock ensues after the xa_recover call is issued against
    an MQ Server resource at recovery time.  This is because the MQ
    Server resource calls back into the Control Region and
    attempts to lock all known transactions to generate a list of
    transactions needed to respond to the xa_recover call.  The
    IN-COMMIT transaction being committed is already locked so the
    xa_recover call hangs waiting to lock the IN-COMMIT
    transaction which will not be unlocked until the transaction
    commits.
    

Problem conclusion

  • This APAR introduces a new custom property of the transaction
    service:
    
    ZOS_RECOVER_BEFORE_COMMIT (default true)
    
    The default value "true" has no effect.  When set to "false"
    this property prevents the xa_recover call prior to xa_commit
    or xa_rollback during the processing of transactions at
    recovery time.  This avoids the deadlock described by this APAR.
    
    To set this property:
    
    1. Login to the Administrative Console.
    2. Select Servers -> Application Servers -> <server> ->
       Container Services -> Transaction Service.
    3. Select the Configuration tab.
    4. Select Custom Properties.
    5. Click new.
    6. Enter ZOS_RECOVER_BEFORE_COMMIT as the name, and "false" as
       the value.
    7. Click Ok.
    8. Save changes.
    9. Restart the server.
    
    All up to date DB2 drivers have an auto-recover feature which
    makes xa_recover prior to xa_commit or xa_rollback
    unnecessary.  If unsure, or it is not clear from the
    documentation for the version of the DB2 driver used, consult
    IBM DB2 support for final clarification on whether setting
    this property is safe for servers hosting applications that
    access DB2.
    
    APAR PM75705 requires changes to documentation.
    
    NOTE: Periodically, we refresh the documentation on our Web
    site, so the changes might have been made before you read this
    text. To access the latest on-line documentation, go to the
    product library page at:
    
    http://www.ibm.com/software/webservers/appserv/library
    
    The following changes to the WebSphere Application Server
    Version 7.0 Information Center will be made available in
    September 2013.
    
    The topic "Transaction service custom properties" will be
    updated to include the following description of the
    ZOS_RECOVER_BEFORE_COMMIT custom property:
    
    ZOS_RECOVER_BEFORE_COMMIT
    
    Specifying this property prevents a deadlock from occurring
    after an xa_recover call is issued against an MQ Server
    resource at recovery time. The MQ Server uses JCA 1.5
    transaction in-flow interfaces as part of its normal
    operation. These interfaces enable MQ Server Bus members to
    coordinate WMQ and Service Integration Bus resources using
    WebSphere Application Server transactions. During the commit
    of any transaction at recovery time, xa_recover is called to
    ensure some older DB2 drivers do not return XAER_NOTA even if
    the transaction exists when the xa_commit or xa_rollback call
    is issued.
    
    This deadlock occurs because the MQ Server resource issues
    calls back to the controller and attempts to lock all known
    transactions so that the MQ Server resource can generate a
    list of transactions that need to respond to the xa_recover
    call. However, because the IN-COMMIT transaction being
    committed is already locked, the xa_recover call waits
    indefinitely to lock the IN-COMMIT transaction because that
    transaction will not be unlocked until the transaction commits.
    
    Setting this property to FALSE ensures that during the
    processing of transactions at recovery time, the xa_recover
    call is not issued before an xa_commit or xa_rollback call.
    Avoid Trouble: All currently supported DB2 drivers have an
    auto-recover feature which makes it unnecessary to issue an
    xa_recover calls prior to issuing an xa_commit or xa_rollback
    call. If, after reading the documentation for your DB2 driver,
    you are not sure whether your DB2 driver includes the
    auto-recover feature, contact IBM DB2 support for final
    clarification on whether setting this property is safe to use
    for servers hosting applications that access DB2.
    Table 8: ZOS_RECOVER_BEFORE_COMMIT custom property
    
    Data Type                    Boolean
    Accepted Values          TRUE, FALSE
    Default                          TRUE
    
    APAR PM75705 is currently targeted for inclusion in Fix Packs
    6.1.0.47, 8.0.0.6 and 8.5.0.2 of WebSphere Application Server.
    
    The fix for the APAR will be delivered as sysrouted APAR PM88418
    in WebSphere Application Server V7.0.
    
    Please refer to URL:
    //www.ibm.com/support/docview.wss?rs=404&uid=swg27006970
    for Fix Pack availability.
    

Temporary fix

Comments

APAR Information

  • APAR number

    PM75705

  • Reported component name

    WEBSPHERE FOR Z

  • Reported component ID

    5655I3500

  • Reported release

    610

  • Status

    CLOSED PER

  • PE

    NoPE

  • HIPER

    NoHIPER

  • Special Attention

    NoSpecatt

  • Submitted date

    2012-10-23

  • Closed date

    2013-01-09

  • Last modified date

    2013-10-04

  • APAR is sysrouted FROM one or more of the following:

  • APAR is sysrouted TO one or more of the following:

    PM80626 PM88418

Fix information

  • Fixed component name

    WEBSPHERE FOR Z

  • Fixed component ID

    5655I3500

Applicable component levels

  • R610 PSY UK97031

       UP13/09/07 P F309

Fix is available

  • Select the PTF appropriate for your component level. You will be required to sign in. Distribution on physical media is not available in all countries.



Document information

More support for: WebSphere Application Server for z/OS
General

Software version: 6.1

Reference #: PM75705

Modified date: 04 October 2013