IBM Support

PM55680: HIGH NUMBER OF EXCEPTIONDELIVERY### THREADS IN WEBSPHERE APP SERVER SERVANT REGION

Fixes are available

7.0.0.23: WebSphere Application Server V7.0 Fix Pack 23
8.0.0.4: WebSphere Application Server V8.0 Fix Pack 4
7.0.0.25: WebSphere Application Server V7.0 Fix Pack 25
8.0.0.5: WebSphere Application Server V8.0 Fix Pack 5
7.0.0.27: WebSphere Application Server V7.0 Fix Pack 27
8.0.0.6: WebSphere Application Server V8.0 Fix Pack 6
7.0.0.29: WebSphere Application Server V7.0 Fix Pack 29
8.0.0.7: WebSphere Application Server V8.0 Fix Pack 7
6.1.0.47: WebSphere Application Server V6.1 Fix Pack 47
8.0.0.8: WebSphere Application Server V8.0 Fix Pack 8
7.0.0.31: WebSphere Application Server V7.0 Fix Pack 31
7.0.0.27: Java SDK 1.6 SR13 FP2 Cumulative Fix for WebSphere Application Server
7.0.0.33: WebSphere Application Server V7.0 Fix Pack 33
8.0.0.9: WebSphere Application Server V8.0 Fix Pack 9
7.0.0.35: WebSphere Application Server V7.0 Fix Pack 35
8.0.0.10: WebSphere Application Server V8.0 Fix Pack 10
7.0.0.37: WebSphere Application Server V7.0 Fix Pack 37
8.0.0.11: WebSphere Application Server V8.0 Fix Pack 11
7.0.0.39: WebSphere Application Server V7.0 Fix Pack 39
8.0.0.12: WebSphere Application Server V8.0 Fix Pack 12
7.0.0.41: WebSphere Application Server V7.0 Fix Pack 41
8.0.0.13: WebSphere Application Server V8.0 Fix Pack 13
7.0.0.43: WebSphere Application Server V7.0 Fix Pack 43
8.0.0.14: WebSphere Application Server V8.0 Fix Pack 14
7.0.0.45: WebSphere Application Server V7.0 Fix Pack 45
8.0.0.15: WebSphere Application Server V8.0 Fix Pack 15
6.1.0.45: Java SDK 1.5 SR14 Cumulative Fix for WebSphere Application Server
6.1.0.47: Java SDK 1.5 SR16 Cumulative Fix for WebSphere Application Server
7.0.0.23: Java SDK 1.6 SR10 FP1 Cumulative Fix for WebSphere
7.0.0.25: Java SDK 1.6 SR11 Cumulative Fix for WebSphere Application Server
7.0.0.27: Java SDK 1.6 SR12 Cumulative Fix for WebSphere Application Server
7.0.0.29: Java SDK 1.6 SR13 FP2 Cumulative Fix for WebSphere Application Server
7.0.0.45: Java SDK 1.6 SR16 FP60 Cumulative Fix for WebSphere Application Server
7.0.0.31: Java SDK 1.6 SR15 Cumulative Fix for WebSphere Application Server
7.0.0.35: Java SDK 1.6 SR16 FP1 Cumulative Fix for WebSphere Application Server
7.0.0.37: Java SDK 1.6 SR16 FP3 Cumulative Fix for WebSphere Application Server
7.0.0.39: Java SDK 1.6 SR16 FP7 Cumulative Fix for WebSphere Application Server
7.0.0.41: Java SDK 1.6 SR16 FP20 Cumulative Fix for WebSphere Application Server
7.0.0.43: Java SDK 1.6 SR16 FP41 Cumulative Fix for WebSphere Application Server
Obtain the fix for this APAR.

Subscribe

You can track all active APARs for this component.

APAR status

  • Closed as program error.

Error description

  • The WebSphere Application Server V6.1 servant region terminates
    due to OutOfMemoryError due to native storage exhaustion.
    Several S878 Abends are logged. Javacores generated at the time
    of the error, and also a TDUMP generated from the error shows
    505 instances of com/ibm/mq/jms/ExceptionDeliveryThread.
    Threads are named "ExceptionDelivery0" through
    "ExceptionDelivery504" inclusive.
    
    There are 505 instances of object
    com/ibm/mq/jms/ExceptionDeliveryThread
    
    For each ExceptionDeliveryThread object there is an
    com/ibm/mq/jms/MQXAConnection object.
    
    504 of the MQXAConnection objects have a state of "stopped",
    and one has a state of "started".
    
    There are also 505 instances of object
    com/ibm/ejs/jms/JMSManagedConnection.
    
    In each case, the JMSMangedConnection object references the
    MQXAConnection, which references the ExceptionDeliveryThread.
    
    Since the JMSManagedConnection is registered as the exception
    listener, the ExceptionDeliveryThread has a reference back to
    the JMSManagedConnection.
    
    This looks like the JMSManagedConnections have been orphaned
    (leaked). The ExcpetionDeliveryThread objects haven't been
    cleaned up because the MQXAConnection hasn't been closed. Since
    the ExceptionDeliveryThread object is referenced from its own
    thread instance, this keeps all these objects from being garbage
    collected.
    
    SYSPRINT shows XARecoveryAgentImpl runs frequently in this
    servant to recover XA resources:
    
    Trace: 2011/10/21 10:17:48.738 01 t=9A50B8 c=UNK key=P8
    (13007002)
    ThreadId: 00000098
    FunctionName: com.ibm.ws390.tx.XARecoveryAgentImpl
    SourceId: com.ibm.ws390.tx.XARecoveryAgentImpl
    Category: FINEST
    ExtendedMessage: There are 460 resources to recover
    ...
    Trace: 2011/10/21 10:19:47.250 01 t=9A50B8 c=UNK key=P8
    (13007002)
    ThreadId: 00000099
    FunctionName: com.ibm.ws390.tx.XARecoveryAgentImpl
    SourceId: com.ibm.ws390.tx.XARecoveryAgentImpl
    Category: FINEST
    ExtendedMessage: There are 460 resources to recover
    ...
    
    Near to the servant region termination, this is logged:
    
    Trace: 2011/10/21 11:27:56.211 01 t=947470 c=UNK key=P8
    (13007002)
    ThreadId: 000000bc
    FunctionName: com.ibm.ws.Transaction.JTA.XARecoveryData
    SourceId: com.ibm.ws.Transaction.JTA.XARecoveryData
    Category: SEVERE
    ExtendedMessage: BBOO0220E: WTRN0005W: The XAResource for a
    transaction participant could not be recreated and
    transaction recovery may not be able to complete properly.
    The resource was J2CXAResourceInfo:
    cfName = JMS$ADTQCFODS$JMSManagedConnection@0
    configProps = [Deployed Resource Adapter Properties]
    TransactionResourceRegistration   java.lang.String
    dynamic   InactiveConnectionSupport   java.lang.Boolean
    true   mcfClassname   java.lang.String
    com.ibm.ejs.jms.WSJMSManagedConnectionFactory
    mcfConnectorName   java.lang.String   X
    mcfObjectType   java.lang.Integer   9
    dd = [Resource Adapter DD]
    ManagedConnectionFactory Class Name:
    com.ibm.ejs.jms.JMSManagedSessionFactory
    Connection Factory Interface Name:
    NO_CF_INTERFACE_SPECIFIED
    Transaction Support:                 2
    Reauthentication Support:            false
    dsProps = {SENDEXITINIT=MQ Bindings Send Exit, PINT=5000,
    QMGR=x, UCP=1, DESC=Project Connect Q - Connection
    Factory, BVER=1, CL=SAFE, MSEL=1, SRI=60000,
    CHAN=x, SSUBS=NO, TRAN=1, RINT=5000,
    RECEXITINIT=MQ Bindings Receive Exit,
    SENDEXIT=
    com.ibm.ws.sca.internal.mq.exit.MQInternalSendExitImpl,
    SS=2, MRET=1, MAPNAMESTYLE=COMPATIBLE, DAUTH=BASIC,
    PORT=x, CLINT=3600000, PAI=25, FIQ=1,
    RECEXIT=
    com.ibm.ws.sca.internal.mq.exit.MQInternalReceiveExitImpl,
    HOST=x.x.x.x, MCAST=DISABLED}
    mcfProps = null
    mmProps = null
    mbeanProps = null
    cmConfig = [Resource-ref CMConfigData key items]
    res-auth:                 1 (APPLICATION)
    res-isolation-level:      0 (TRANSACTION_NONE)
    res-sharing-scope:        false (UNSHAREABLE)
    loginConfigurationName:   null
    loginConfigProperties:    null
    [Resource-ref non-key items]
    res-resolution-control:   999
    (undefined)
    res ref or CMP bean name: null
    Database pool properties: null
    primeID:                  0
    isCMP1_x:                 false (not CMP1.x)
    isJMS:                 true (JMS)
    recoveryClasspath = null.
    The exception stack trace follows:
    java.lang.OutOfMemoryError: Failed to create a thread:
    retVal -1073741830, errno 132
    at java.lang.Thread.startImpl(Native Method)
    at java.lang.Thread.start(Thread.java:980)
    at com.ibm.mq.jms.ExceptionDeliveryThread.<init>
    (ExceptionDeliveryThread.java:122)
    at com.ibm.mq.jms.MQConnection.setExceptionListener(
    MQConnection.java:1460)
    at com.ibm.mq.jms.MQXAConnection.setExceptionListener(
    MQXAConnection.java:329)
    at com.ibm.ejs.jms.JMSManagedConnection.<init>(
    JMSManagedConnection.java:342)
    at com.ibm.ejs.jms.JMSManagedConnectionFactory.
    createUnifiedManagedConnection(
    JMSManagedConnectionFactory.java:1270)
    at com.ibm.ejs.jms.JMSManagedConnectionFactory.
    createManagedConnection(
    JMSManagedConnectionFactory.java:1287)
    at com.ibm.ejs.jms.JMSManagedSessionFactory.
    getManagedConnection(
    JMSManagedSessionFactory.java:821)
    at com.ibm.ejs.jms.JMSManagedSessionFactory.
    getPhysicalConnection(
    JMSManagedSessionFactory.java:790)
    at com.ibm.ejs.jms.JMSManagedSession.<init>(
    JMSManagedSession.java:205)
    at com.ibm.ejs.jms.JMSManagedSessionFactory.
    createManagedSession(JMSManagedSessionFactory.java:652)
    at com.ibm.ejs.jms.JMSManagedSessionFactory.
    createManagedConnection(JMSManagedSessionFactory.java:430)
    at com.ibm.ejs.j2c.J2CXAResourceFactory$1.run(
    J2CXAResourceFactory.java:323)
    at com.ibm.ws.security.util.AccessController.doPrivileged(
    AccessController.java:118)
    at com.ibm.ejs.j2c.J2CXAResourceFactory.getXAResource(
    J2CXAResourceFactory.java:132)
    at com.ibm.ws.Transaction.JTA.XARecoveryData.getXARminst(
    XARecoveryData.java:544)
    at com.ibm.ws.Transaction.JTA.XARecoveryData.recover(
    XARecoveryData.java:669)
    at com.ibm.ws390.tx.XARecoveryAgentImpl.
    rollbackUnknownTransactions(XARecoveryAgentImpl.java:716)
    at com.ibm.ws390.tx.XARecoveryAgentImpl$
    XARecoveryAgentThread.run(XARecoveryAgentImpl.java:351)
    
    ADDITIONAL SYMPTOMS:
    There is an MQ connection leak in addition to the thread leak.
    On V6.1, the out-of-storage condition caused by the thread leak
    will likely cause the servant region to ABEND before the MQ
    connections hit max. However, on WebSphere Application Server
    V7 there is no thread leak and this bug can lead to
    +CSQX489E LPW1 CSQXRESP Maximum instance limit 300, exceeded,
    channel ABC.DDDEFFF.CLCLCLCL, connection 192.168.1.230
    

Local fix

Problem summary

  • ****************************************************************
    * USERS AFFECTED:  All users of IBM WebSphere Application      *
    *                  Server V6.1 and V8.0                        *
    ****************************************************************
    * PROBLEM DESCRIPTION: During transaction recovery a WMQ       *
    *                      ExceptionDeliveryThread is leaked for   *
    *                      each WMQ resource that needs recovery   *
    *                      by the WebSphere Transaction Manager.   *
    ****************************************************************
    * RECOMMENDATION:                                              *
    ****************************************************************
    During transaction recovery of an WMQ resource a
    MQExceptionDelivery Thread is started and never cleaned up. If
    transaction recovery fails for any reason then recovery is
    reattempted for all resources regardless as to whether these
    resources sucessfully recovered or not. This will potentially
    lead to many MQExceptionDeliveryThreads being created and
    causing an OutOfMemoryError.
    This leak is present on all platforms but due to the way
    transaction recovery occurs on zOS then it is more likely to
    reach an OutOfMemoryError condition on this platform.
    

Problem conclusion

  • This APAR fixes the problem by closing the MQ Connection once
    the MQ resource has finished recovery.
    
    APAR PM55680 is currently targeted for inclusion in Service
    Level (Fix Pack) 6.1.0.45 of WebSphere Application Server V6.1
    and Fix Pack 8.0.0.4 of WebSphere Application Server V8.0.
    
    The fix will also be delivered as sysrouted APAR PM55965 in
    WebSphere Application Server V7.0 Fix Pack 7.0.0.23.
    
    Please refer to URL:
    //www.ibm.com/support/docview.wss?rs=404&uid=swg27006970
    for Fix Pack availability.
    

Temporary fix

Comments

APAR Information

  • APAR number

    PM55680

  • Reported component name

    WEBSPHERE FOR Z

  • Reported component ID

    5655I3500

  • Reported release

    610

  • Status

    CLOSED PER

  • PE

    NoPE

  • HIPER

    NoHIPER

  • Special Attention

    NoSpecatt

  • Submitted date

    2012-01-10

  • Closed date

    2012-02-16

  • Last modified date

    2012-10-01

  • APAR is sysrouted FROM one or more of the following:

  • APAR is sysrouted TO one or more of the following:

    PM55965

Fix information

  • Fixed component name

    WEBSPHERE FOR Z

  • Fixed component ID

    5655I3500

Applicable component levels

  • R610 PSY UK81646

       UP12/09/23 P F209

Fix is available

  • Select the PTF appropriate for your component level. You will be required to sign in. Distribution on physical media is not available in all countries.



Document information

More support for: WebSphere Application Server for z/OS
General

Software version: 6.1

Reference #: PM55680

Modified date: 01 October 2012