PM55680: HIGH NUMBER OF EXCEPTIONDELIVERY### THREADS IN WEBSPHERE APP SERVER SERVANT REGION

Fixes are available

APAR status

Closed as program error.

Error description

The WebSphere Application Server V6.1 servant region terminates
due to OutOfMemoryError due to native storage exhaustion.
Several S878 Abends are logged. Javacores generated at the time
of the error, and also a TDUMP generated from the error shows
505 instances of com/ibm/mq/jms/ExceptionDeliveryThread.
Threads are named "ExceptionDelivery0" through
"ExceptionDelivery504" inclusive.

There are 505 instances of object
com/ibm/mq/jms/ExceptionDeliveryThread

For each ExceptionDeliveryThread object there is an
com/ibm/mq/jms/MQXAConnection object.

504 of the MQXAConnection objects have a state of "stopped",
and one has a state of "started".

There are also 505 instances of object
com/ibm/ejs/jms/JMSManagedConnection.

In each case, the JMSMangedConnection object references the
MQXAConnection, which references the ExceptionDeliveryThread.

Since the JMSManagedConnection is registered as the exception
listener, the ExceptionDeliveryThread has a reference back to
the JMSManagedConnection.

This looks like the JMSManagedConnections have been orphaned
(leaked). The ExcpetionDeliveryThread objects haven't been
cleaned up because the MQXAConnection hasn't been closed. Since
the ExceptionDeliveryThread object is referenced from its own
thread instance, this keeps all these objects from being garbage
collected.

SYSPRINT shows XARecoveryAgentImpl runs frequently in this
servant to recover XA resources:

Trace: 2011/10/21 10:17:48.738 01 t=9A50B8 c=UNK key=P8
(13007002)
ThreadId: 00000098
FunctionName: com.ibm.ws390.tx.XARecoveryAgentImpl
SourceId: com.ibm.ws390.tx.XARecoveryAgentImpl
Category: FINEST
ExtendedMessage: There are 460 resources to recover
...
Trace: 2011/10/21 10:19:47.250 01 t=9A50B8 c=UNK key=P8
(13007002)
ThreadId: 00000099
FunctionName: com.ibm.ws390.tx.XARecoveryAgentImpl
SourceId: com.ibm.ws390.tx.XARecoveryAgentImpl
Category: FINEST
ExtendedMessage: There are 460 resources to recover
...

Near to the servant region termination, this is logged:

Trace: 2011/10/21 11:27:56.211 01 t=947470 c=UNK key=P8
(13007002)
ThreadId: 000000bc
FunctionName: com.ibm.ws.Transaction.JTA.XARecoveryData
SourceId: com.ibm.ws.Transaction.JTA.XARecoveryData
Category: SEVERE
ExtendedMessage: BBOO0220E: WTRN0005W: The XAResource for a
transaction participant could not be recreated and
transaction recovery may not be able to complete properly.
The resource was J2CXAResourceInfo:
cfName = JMS$ADTQCFODS$JMSManagedConnection@0
configProps = [Deployed Resource Adapter Properties]
TransactionResourceRegistration   java.lang.String
dynamic   InactiveConnectionSupport   java.lang.Boolean
true   mcfClassname   java.lang.String
com.ibm.ejs.jms.WSJMSManagedConnectionFactory
mcfConnectorName   java.lang.String   X
mcfObjectType   java.lang.Integer   9
dd = [Resource Adapter DD]
ManagedConnectionFactory Class Name:
com.ibm.ejs.jms.JMSManagedSessionFactory
Connection Factory Interface Name:
NO_CF_INTERFACE_SPECIFIED
Transaction Support:                 2
Reauthentication Support:            false
dsProps = {SENDEXITINIT=MQ Bindings Send Exit, PINT=5000,
QMGR=x, UCP=1, DESC=Project Connect Q - Connection
Factory, BVER=1, CL=SAFE, MSEL=1, SRI=60000,
CHAN=x, SSUBS=NO, TRAN=1, RINT=5000,
RECEXITINIT=MQ Bindings Receive Exit,
SENDEXIT=
com.ibm.ws.sca.internal.mq.exit.MQInternalSendExitImpl,
SS=2, MRET=1, MAPNAMESTYLE=COMPATIBLE, DAUTH=BASIC,
PORT=x, CLINT=3600000, PAI=25, FIQ=1,
RECEXIT=
com.ibm.ws.sca.internal.mq.exit.MQInternalReceiveExitImpl,
HOST=x.x.x.x, MCAST=DISABLED}
mcfProps = null
mmProps = null
mbeanProps = null
cmConfig = [Resource-ref CMConfigData key items]
res-auth:                 1 (APPLICATION)
res-isolation-level:      0 (TRANSACTION_NONE)
res-sharing-scope:        false (UNSHAREABLE)
loginConfigurationName:   null
loginConfigProperties:    null
[Resource-ref non-key items]
res-resolution-control:   999
(undefined)
res ref or CMP bean name: null
Database pool properties: null
primeID:                  0
isCMP1_x:                 false (not CMP1.x)
isJMS:                 true (JMS)
recoveryClasspath = null.
The exception stack trace follows:
java.lang.OutOfMemoryError: Failed to create a thread:
retVal -1073741830, errno 132
at java.lang.Thread.startImpl(Native Method)
at java.lang.Thread.start(Thread.java:980)
at com.ibm.mq.jms.ExceptionDeliveryThread.<init>
(ExceptionDeliveryThread.java:122)
at com.ibm.mq.jms.MQConnection.setExceptionListener(
MQConnection.java:1460)
at com.ibm.mq.jms.MQXAConnection.setExceptionListener(
MQXAConnection.java:329)
at com.ibm.ejs.jms.JMSManagedConnection.<init>(
JMSManagedConnection.java:342)
at com.ibm.ejs.jms.JMSManagedConnectionFactory.
createUnifiedManagedConnection(
JMSManagedConnectionFactory.java:1270)
at com.ibm.ejs.jms.JMSManagedConnectionFactory.
createManagedConnection(
JMSManagedConnectionFactory.java:1287)
at com.ibm.ejs.jms.JMSManagedSessionFactory.
getManagedConnection(
JMSManagedSessionFactory.java:821)
at com.ibm.ejs.jms.JMSManagedSessionFactory.
getPhysicalConnection(
JMSManagedSessionFactory.java:790)
at com.ibm.ejs.jms.JMSManagedSession.<init>(
JMSManagedSession.java:205)
at com.ibm.ejs.jms.JMSManagedSessionFactory.
createManagedSession(JMSManagedSessionFactory.java:652)
at com.ibm.ejs.jms.JMSManagedSessionFactory.
createManagedConnection(JMSManagedSessionFactory.java:430)
at com.ibm.ejs.j2c.J2CXAResourceFactory$1.run(
J2CXAResourceFactory.java:323)
at com.ibm.ws.security.util.AccessController.doPrivileged(
AccessController.java:118)
at com.ibm.ejs.j2c.J2CXAResourceFactory.getXAResource(
J2CXAResourceFactory.java:132)
at com.ibm.ws.Transaction.JTA.XARecoveryData.getXARminst(
XARecoveryData.java:544)
at com.ibm.ws.Transaction.JTA.XARecoveryData.recover(
XARecoveryData.java:669)
at com.ibm.ws390.tx.XARecoveryAgentImpl.
rollbackUnknownTransactions(XARecoveryAgentImpl.java:716)
at com.ibm.ws390.tx.XARecoveryAgentImpl$
XARecoveryAgentThread.run(XARecoveryAgentImpl.java:351)

ADDITIONAL SYMPTOMS:
There is an MQ connection leak in addition to the thread leak.
On V6.1, the out-of-storage condition caused by the thread leak
will likely cause the servant region to ABEND before the MQ
connections hit max. However, on WebSphere Application Server
V7 there is no thread leak and this bug can lead to
+CSQX489E LPW1 CSQXRESP Maximum instance limit 300, exceeded,
channel ABC.DDDEFFF.CLCLCLCL, connection 192.168.1.230

Local fix

Problem summary

****************************************************************
* USERS AFFECTED:  All users of IBM WebSphere Application      *
*                  Server V6.1 and V8.0                        *
****************************************************************
* PROBLEM DESCRIPTION: During transaction recovery a WMQ       *
*                      ExceptionDeliveryThread is leaked for   *
*                      each WMQ resource that needs recovery   *
*                      by the WebSphere Transaction Manager.   *
****************************************************************
* RECOMMENDATION:                                              *
****************************************************************
During transaction recovery of an WMQ resource a
MQExceptionDelivery Thread is started and never cleaned up. If
transaction recovery fails for any reason then recovery is
reattempted for all resources regardless as to whether these
resources sucessfully recovered or not. This will potentially
lead to many MQExceptionDeliveryThreads being created and
causing an OutOfMemoryError.
This leak is present on all platforms but due to the way
transaction recovery occurs on zOS then it is more likely to
reach an OutOfMemoryError condition on this platform.

Problem conclusion

This APAR fixes the problem by closing the MQ Connection once
the MQ resource has finished recovery.

APAR PM55680 is currently targeted for inclusion in Service
Level (Fix Pack) 6.1.0.45 of WebSphere Application Server V6.1
and Fix Pack 8.0.0.4 of WebSphere Application Server V8.0.

The fix will also be delivered as sysrouted APAR PM55965 in
WebSphere Application Server V7.0 Fix Pack 7.0.0.23.

Please refer to URL:
//www.ibm.com/support/docview.wss?rs=404&uid=swg27006970
for Fix Pack availability.

Temporary fix

Comments

APAR Information

APAR number
PM55680
Reported component name
WEBSPHERE FOR Z
Reported component ID
5655I3500
Reported release
610
Status
CLOSED PER
PE
NoPE
HIPER
NoHIPER
Special Attention
NoSpecatt
Submitted date
2012-01-10
Closed date
2012-02-16
Last modified date
2012-10-01

APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:

PM55965

Fix information

Fixed component name
WEBSPHERE FOR Z
Fixed component ID
5655I3500

Applicable component levels

R610 PSY UK81646
UP12/09/23 P F209

Fix is available

Select the PTF appropriate for your component level. You will be required to sign in. Distribution on physical media is not available in all countries.

[{"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SS7K4U","label":"WebSphere Application Server for z\/OS"},"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"6.1","Line of Business":{"code":"LOB45","label":"Automation"}}]

Document Information

Modified date:
28 October 2021

Tips

PM55680: HIGH NUMBER OF EXCEPTIONDELIVERY### THREADS IN WEBSPHERE APP SERVER SERVANT REGION

Fixes are available

Subscribe

APAR status

Closed as program error.

Error description

Local fix

Problem summary

Problem conclusion

Temporary fix

Comments

APAR Information

APAR number

Reported component name

Reported component ID

Reported release

Status

PE

HIPER

Special Attention

Submitted date

Closed date

Last modified date

APAR is sysrouted FROM one or more of the following:

APAR is sysrouted TO one or more of the following:

Fix information

Fixed component name

Fixed component ID

Applicable component levels

R610 PSY UK81646

Fix is available

Select the PTF appropriate for your component level. You will be required to sign in. Distribution on physical media is not available in all countries.

Document Information

Share your feedback

Need support?