IT37502: WebSphere Application Server transaction recovery fails when activation specs are configured to use BINDINGS_THEN_CLIENT

APAR status

• Closed as program error.

Error description

• WebSphere Application Server 9.0.x and IBM MQ are installed on
  separate machines.
  
  An IBM MQ messaging provider activation specification has been
  defined within the application server, and is configured to
  connect to a queue manager using the BINDINGS_THEN_CLIENT
  transport.
  
  While the activation specification is processing messages, the
  application server is stopped unexpectedly. When the application
  server restarts, it tries to recover the XA transactions that
  the activation specification was involved in at the time that
  the application server stopped. However, the transaction
  recovery fails and the following message is written to the
  application server's SystemOut.log file:
  
  WTRN0141I: Recovered transaction (tid=2044193076) commit of xid
  0000017A3E7525A40000000179D3A864F0C16576CBF488D9DEBDBED0AFA31020
  65E959CB:0000017A3E7525A40000000179D3A864F0C16576CBF488D9DEBDBED
  0AFA3102065E959CB000000010000000000000000000000000001 with
  XAResource:
  cells/LAPTOP-R910IGN3Node18Cell/resources.xml#J2CResourceAdapter
  _1612870301746 resulted in XAER_RMFAIL
  

Local fix

• Update the transport of MQ connection factory or Activation
  Spec to 'Client'.
  

Problem summary

• ****************************************************************
  USERS AFFECTED:
  This issue affects users of the WebSphere Application Server
  9.0.x IBM MQ messaging provider, who:
  
  - Have the application server installed on a different machine
  to an IBM MQ queue manager.
  - And have activation specifications defined within the
  application server that are configured to connect to the queue
  manager using the default transport type of
  BINDINGS_THEN_CLIENT.
  
  
  Platforms affected:
  MultiPlatform
  
  ****************************************************************
  PROBLEM DESCRIPTION:
  When creating an IBM MQ messaging provider activation
  specification in WebSphere Application Server, the default
  transport mode is BINDINGS_THEN_CLIENT. When an activation
  specification which is configured to use this transport mode is
  started, then the MQ resource adapter (the component of
  WebSphere Application Server that handles all of the
  communication with IBM MQ queue managers) will initially try to
  connect to the specified queue manager using the BINDINGS
  transport. If the connection attempt fails, then the MQ resource
  adapter will try to connect to it using the CLIENT transport. If
  that attempt also fails, then the MQ resource adapter will throw
  a JMSException back to the application server and the activation
  specification will not start.
  
  Now, if a message-driven bean (MDB) application is:
  
  - Deployed into WebSphere Application Server.
  - And is configured to use an IBM MQ messaging provider
  activation specification to consume messages from a queue hosted
  on a queue manager.
  - And is also configured to use XA transactions.
  
  then whenever a transaction is started, the application server
  will store a reference to the activation specification to its
  transaction logs. If the XA transaction needs to be recovered
  for some reason, then the application server will use the
  details of the activation specification stored in its
  transaction logs to reconnect to the queue manager and then
  recover the transaction.
  
  
  When the issue reported in this APAR occurred, the application
  server had stopped unexpectedly:
  
  - After an xa_commit() had been flowed to the queue manager to
  complete an XA transaction involving an activation specification
  that was configured to use the BINDINGS_THEN_CLIENT transport.
  - And before the application server had forced a "deletion
  record" to its transaction logs.
  
  When the application server restarted, the following steps
  occurred:
  
  - The application server checked its transaction logs and found
  an entry for the XA transaction.
  - It then called into the MQ resource adapter, asking it to use
  the activation specification definition associated with the XA
  transaction to connect to the queue manager and retry the
  xa_commit() operation.
  
  - Initially, the MQ resource adapter tried to connect to the
  queue manager using the BINDINGS transport. This attempt failed
  with the following JMSException:
  
  JMSFMQ6312: An exception occurred in the Java(tm) MQI. The
  Java(tm) MQI has thrown an exception describing the problem. See
  the linked exception for further information.
  ...
  Caused by: com.ibm.mq.jmqi.JmqiException: CC=2;RC=2495;AMQ8568:
  The native JNI library 'mqjbnd64' was not found. For a client
  installation this is expected. [3=mqjbnd64]
  ...
  Caused by: java.lang.UnsatisfiedLinkError: mqjbnd64 (Not found
  in java.library.path)
  	at
  java.lang.ClassLoader.loadLibraryWithPath(ClassLoader.java:1459)
  	at
  java.lang.ClassLoader.loadLibraryWithClassLoader(ClassLoader.jav
  a:1410)
  	at java.lang.System.loadLibrary(System.java:589)
  	at com.ibm.mq.jmqi.local.LocalMQ.loadLib(LocalMQ.java:1112)
  	... 31 more
  
  
  which the MQ resource adapter stored in an internal list.
  
  - Next, the MQ resource adapter tried to connect to the queue
  manager using the CLIENT transport. This attempt was successful.
  - The MQ resource adapter then flowed an xa_commit() call to the
  queue manager to complete the transaction.
  - Because the queue manager had already committed the
  transaction, it returned XA error code XAER_NOTA (-4) back to
  the MQ resource adapter.
  - When the MQ resource adapter received this XA error code from
  the queue manager, it created the following internal
  JMSExcetion:
  
  [com.ibm.msg.client.jms.DetailedJMSException: JMSFMQ6312: An
  exception occurred in the Java(tm) MQI. The Java(tm) MQI has
  thrown an exception describing the problem. See the linked
  exception for further information.,
  javax.transaction.xa.XAException: The method 'xa_commit' has
  failed with errorCode '-4'.]
  
  and added it to the internal list.
  
  - The MQ resource adapter now processed the exceptions in the
  list, to determine which one to return back to the application
  server.
  - The correct exception to return was the second one, as it
  indicated that the queue manager had already completed all of
  the work for the transaction that was being recovered. However,
  the MQ resource adapter only ever checked the first exception in
  the list. Because this exception represented a failure to
  connect to the queue manager, the MQ resource adapter
  constructed an XAException containing:
  
      - The XA error code -7 (XAER_RMFAIL).
      - And the first exception as a linked exception.
  
  This exception was then returned back to the application server.
  - When the application server received the exception, it
  determined that the transaction recovery had failed. This caused
  it to write the message:
  
  WTRN0141I: Recovered transaction (tid=<identifier>) commit of
  xid <transaction identifier>with XAResource:<resource
  identiifer> resulted in XAER_RMFAIL
  
  to it's SystemOut.log file, and then wait for a period of time
  before trying to recover the transaction again.
  
  
  Whenever the application server tried to recover the transaction
  again, the same sequence of events occurred. As a result, the
  transaction could never be recovered, and the entry for it was
  never removed from the application server's transaction logs.
  

Problem conclusion

• In order to resolve this issue, the MQ resource adapter has been
  updated to check both exceptions in the list when determining
  what to return back to the application server when recovering a
  transaction associated with an activation specification that has
  been configured to use the BINDINGS_THEN_CLIENT transport,
  rather than just the first one.
  
  This ensures that the correct exception is passed back the
  application server for the current transaction recovery
  scenario, which in turn allows the recovery to continue as
  expected.
  
  ---------------------------------------------------------------
  The fix is targeted for delivery in the following PTFs:
  
  Version    Maintenance Level
  v9.1 LTS   9.1.0.11
  v9.2 LTS   9.2.0.5
  v9.x CD    9.2.5
  
  The latest available maintenance can be obtained from
  'WebSphere MQ Recommended Fixes'
  http://www-1.ibm.com/support/docview.wss?rs=171&uid=swg27006037
  
  If the maintenance level is not yet available information on
  its planned availability can be found in 'WebSphere MQ
  Planned Maintenance Release Dates'
  http://www-1.ibm.com/support/docview.wss?rs=171&uid=swg27006309
  ---------------------------------------------------------------
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  IBM MQ BASE MP

• Fixed component ID

  5724H7271

Applicable component levels