IBM Support

PM11713: JAVA.LANG.OUTOFMEMORYERROR FOLLOWING A HIGH VOLUME OF EJB CALLS PROCESSED BY AN APPSERVER

Fixes are available

PM11713: Out Of Memory exceptions after large spikes in load with large tran..
7.0.0.11: WebSphere Application Server V7.0 Fix Pack 11 for IBM i
7.0.0.11: WebSphere Application Server V7.0 Fix Pack 11 for Windows
7.0.0.11: WebSphere Application Server V7.0 Fix Pack 11 for HP-UX
7.0.0.11: WebSphere Application Server V7.0 Fix Pack 11 for AIX
7.0.0.11: Java SDK 1.6 SR7 Cumulative Fix for WebSphere Application Server
7.0.0.11: WebSphere Application Server V7.0 Fix Pack 11 for Solaris
7.0.0.11: WebSphere Application Server V7.0 Fix Pack 11 for Linux
6.1.0.33: Java SDK 1.5 SR12 FP1 Cumulative Fix for WebSphere
7.0.0.13: WebSphere Application Server V7.0 Fix Pack 13 for AIX
7.0.0.13: WebSphere Application Server V7.0 Fix Pack 13 for HP-UX
7.0.0.13: WebSphere Application Server V7.0 Fix Pack 13 for IBM i
7.0.0.13: WebSphere Application Server V7.0 Fix Pack 13 for Linux
7.0.0.13: WebSphere Application Server V7.0 Fix Pack 13 for Solaris
7.0.0.13: WebSphere Application Server V7.0 Fix Pack 13 for Windows
7.0.0.13: Java SDK 1.6 SR8FP1 Cumulative Fix for WebSphere Application Server
6.1.0.35: Java SDK 1.5 SR12 FP2 Cumulative Fix for WebSphere
7.0.0.15: WebSphere Application Server V7.0 Fix Pack 15 for AIX
7.0.0.15: Java SDK 1.6 SR9 Cumulative Fix for WebSphere Application Server
7.0.0.15: WebSphere Application Server V7.0 Fix Pack 15 for HP-UX
7.0.0.15: WebSphere Application Server V7.0 Fix Pack 15 for IBM i
7.0.0.15: WebSphere Application Server V7.0 Fix Pack 15 for Linux
7.0.0.15: WebSphere Application Server V7.0 Fix Pack 15 for Solaris
7.0.0.15: WebSphere Application Server V7.0 Fix Pack 15 for Windows
6.1.0.37: Java SDK 1.5 SR12 FP3 Cumulative Fix for WebSphere
7.0.0.17: WebSphere Application Server V7.0 Fix Pack 17
7.0.0.17: Java SDK 1.6 SR9 FP1 Cumulative Fix for WebSphere Application Server
7.0.0.19: WebSphere Application Server V7.0 Fix Pack 19
7.0.0.21: WebSphere Application Server V7.0 Fix Pack 21
7.0.0.23: WebSphere Application Server V7.0 Fix Pack 23
7.0.0.25: WebSphere Application Server V7.0 Fix Pack 25
7.0.0.27: WebSphere Application Server V7.0 Fix Pack 27
7.0.0.29: WebSphere Application Server V7.0 Fix Pack 29
6.1.0.47: WebSphere Application Server V6.1 Fix Pack 47
7.0.0.31: WebSphere Application Server V7.0 Fix Pack 31
7.0.0.27: Java SDK 1.6 SR13 FP2 Cumulative Fix for WebSphere Application Server
7.0.0.33: WebSphere Application Server V7.0 Fix Pack 33
7.0.0.35: WebSphere Application Server V7.0 Fix Pack 35
7.0.0.37: WebSphere Application Server V7.0 Fix Pack 37
7.0.0.39: WebSphere Application Server V7.0 Fix Pack 39
7.0.0.41: WebSphere Application Server V7.0 Fix Pack 41
7.0.0.43: WebSphere Application Server V7.0 Fix Pack 43
7.0.0.45: WebSphere Application Server V7.0 Fix Pack 45
6.1.0.39: Java SDK 1.5 SR12 FP4 Cumulative Fix for WebSphere Application Server
6.1.0.41: Java SDK 1.5 SR12 FP5 Cumulative Fix for WebSphere Application Server
6.1.0.43: Java SDK 1.5 SR13 Cumulative Fix for WebSphere Application Server
6.1.0.45: Java SDK 1.5 SR14 Cumulative Fix for WebSphere Application Server
6.1.0.47: Java SDK 1.5 SR16 Cumulative Fix for WebSphere Application Server
7.0.0.19: Java SDK 1.6 SR9 FP2 Cumulative Fix for WebSphere Application Server
7.0.0.21: Java SDK 1.6 SR9 FP2 Cumulative Fix for WebSphere
7.0.0.23: Java SDK 1.6 SR10 FP1 Cumulative Fix for WebSphere
7.0.0.25: Java SDK 1.6 SR11 Cumulative Fix for WebSphere Application Server
7.0.0.27: Java SDK 1.6 SR12 Cumulative Fix for WebSphere Application Server
7.0.0.29: Java SDK 1.6 SR13 FP2 Cumulative Fix for WebSphere Application Server
7.0.0.45: Java SDK 1.6 SR16 FP60 Cumulative Fix for WebSphere Application Server
7.0.0.31: Java SDK 1.6 SR15 Cumulative Fix for WebSphere Application Server
7.0.0.35: Java SDK 1.6 SR16 FP1 Cumulative Fix for WebSphere Application Server
7.0.0.37: Java SDK 1.6 SR16 FP3 Cumulative Fix for WebSphere Application Server
7.0.0.39: Java SDK 1.6 SR16 FP7 Cumulative Fix for WebSphere Application Server
7.0.0.41: Java SDK 1.6 SR16 FP20 Cumulative Fix for WebSphere Application Server
7.0.0.43: Java SDK 1.6 SR16 FP41 Cumulative Fix for WebSphere Application Server

Subscribe

You can track all active APARs for this component.

 

APAR status

  • Closed as program error.

Error description

  • The primary symptom is an OutOfMemoryError.  This may cause
    CNTR0019E or CNTR0020E errors, since the memory that is not
    being cleaned up is EJB cache memory.
    .
    The heapdump will resemble the following.
    com/ibm/ejs/util/cache/BackgroundLruEvictionStrategy, which has
    a
    com/ibm/ejs/util/cache/CacheElementEnumerator,
    which has an array of
    com/ibm/ejs/util/cache/Element objects.
    
    Each com/ibm/ejs/util/cache/Element has a
    com/ibm/ejs/container/activator/TransactionKey, which has a
    com/ibm/ejs/container/ContainerTx, which then has an ArrayList
    which contains thousands of
    com/ibm/ejs/container/ContainerManagedBeanO objects.
    .
    The problem was reported using IBM WebSphere Commerce Suite, so
    the beans in the EJB cache are EJBs from that product.  The
    issue is not with the CommerceSuite beans, but with how the
    WebSphere Application Server cache is handling them.
    .
    The key symptom in terms of the heap is the Array of
    thousands of ContainerManagedBeanO objects tied to a
    CacheElementEnumerator.
    .--------------------------------------------------------
    This shows that the memory that is potentially being leaked is
    tied to the EJB cache that WebSphere AppServer maintains.
    317,803,376 (33%) [176] 5
    com/ibm/ejs/util/cache/BackgroundLruEvictionStrategy 0x72ebe0c8
    317,803,064 (33%) [56] 3
    com/ibm/ejs/util/cache/CacheElementEnumerator 0x8243f630
    317,802,912 (33%) [472] 26 array of
    java/lang/Object 0xa70484b8
    .--------------------------------------
    Here is one of the elements in the EJB Cache.
    238,307,992 (25%) [40] 2 com/ibm/ejs/util/cache/Element
    0xb6ae1778
    238,307,256 (25%) [24] 2
    com/ibm/ejs/container/activator/TransactionKey 0xb6ae17a0
    238,306,896 (25%) [136] 10
    com/ibm/ejs/container/ContainerTx 0x88a960a0
    164,283,672 (17%) [24] 1 java/util/ArrayList
    0x88d8f5c8
    .
    Here is that giant array of beans.  It is directly tied to a
    cached EJB.  The array consists of lots and lots of
    com/ibm/commerce/order/objects/EJSCMPTradingPositionContainerXML
    DefinHomeBean_c498873a beans.
    What this says to me is that the order objects are not
    being cleaned up and are accumulating.  There is an array of
    185,387 ContainerManagedBeanO objects.
    164,283,648 (17%) [794,328] 185,387 array of
    java/lang/Object 0xa4bd5008
    21,504 (0%) [176] 10
    com/ibm/ejs/container/ContainerManagedBeanO 0x761eb0c8
    21,504 (0%) [176] 10
    com/ibm/ejs/container/ContainerManagedBeanO 0x96b5b6e0
    21,504 (0%) [176] 10
    com/ibm/ejs/container/ContainerManagedBeanO 0x9a4ea550
    21,224 (0%) [48] 4
    com/ibm/commerce/order/objects/TradingPositionContainerXMLDefini
    tionBean 0x9a4ea600
    104 (0%) [24] 1 com/ibm/ejs/j2c/HandleList 0x9a4ea538
    825,919,352 (88%) [200] 33
    com/ibm/ejs/container/EJSContainer 0x8240da48
    959,128 (0%) [160] 18
    com/ibm/commerce/order/objects/EJSCMPTradingPositionContainerXML
    DefinHom eBean_c498873a 0x8f0e5d38
    957,072 (0%) [56] 4
    com/ibm/ejs/csi/PoolImplThreadSafe 0x8f0e8070
    1,576 (0%) [248] 15
    com/ibm/ws/pmi/server/modules/BeanModule 0x8f0e59c0
    872 (0%) [80] 16 array of
    java/lang/String 0x7a456348
    128 (0%) [72] 5
    com/ibm/commerce/order/objects/EJSJDBCPersisterCMPTradingPositio
    nContainerXMLDefinitionBean_c498873a 0x8f0c2948
    24 (0%) [24] 2
    com/ibm/ejs/container/activator/OptCEntityActivationStrategy
    0x82495070
    16 (0%) [16] 0
    com/ibm/ejs/container/lock/LockStrategy 0x9326ee08
    21,504 (0%) [176] 10
    com/ibm/ejs/container/ContainerManagedBeanO 0x8f0fe698
    21,224 (0%) [48] 4
    com/ibm/commerce/order/objects/TradingPositionContainerXMLDefini
    tionBean 0x8f0fe748
    104 (0%) [24] 1
    com/ibm/ejs/j2c/HandleList 0x8f0fe680
    825,919,352 (88%) [200] 33
    com/ibm/ejs/container/EJSContainer 0x8240da48
    959,128 (0%) [160] 18
    com/ibm/commerce/order/objects/EJSCMPTradingPositionContainerXML
    DefinHomeBean_c498873a 0x8f0e5d38
    

Local fix

Problem summary

  • ****************************************************************
    * USERS AFFECTED:  All users of IBM WebSphere Application      *
    *                  Server containing large or long-running     *
    *                  transactions.                               *
    ****************************************************************
    * PROBLEM DESCRIPTION: Users can experience Out Of Memory      *
    *                      exceptions following large spikes in    *
    *                      load with large transactions.           *
    ****************************************************************
    * RECOMMENDATION:                                              *
    ****************************************************************
    Out Of Memory issues may result after large spikes in load.
    Analysis of the heap dumps should expose large "inactive"
    transactions with EJB references being held by the
    BackgroundLruEvictionStrategy.
    This APAR is a follow-on APAR to PK98076 that addresses a
    separate issue in the same area of code.
    

Problem conclusion

  • Modified the EJBContainer code to completely clear the copy of
    the cache bucket so that all items will be garbage collected
    prior to the next valid sweep.
    
    The fix for this APAR is currently targeted for inclusion in
    fix packs 7.0.0.11, 6.1.0.33, Feature Pack for EJB3.0
    6.1.0.33, and 6.0.2.43.
    Please refer to the Recommended Updates page for delivery
    information:
    http://www.ibm.com/support/docview.wss?rs=180&uid=swg27004980
    

Temporary fix

Comments

APAR Information

  • APAR number

    PM11713

  • Reported component name

    WEBSPHERE APP S

  • Reported component ID

    5724J0800

  • Reported release

    60A

  • Status

    CLOSED PER

  • PE

    NoPE

  • HIPER

    NoHIPER

  • Special Attention

    NoSpecatt

  • Submitted date

    2010-04-07

  • Closed date

    2010-04-28

  • Last modified date

    2010-04-28

  • APAR is sysrouted FROM one or more of the following:

  • APAR is sysrouted TO one or more of the following:

Fix information

  • Fixed component name

    WEBSPHERE APP S

  • Fixed component ID

    5724J0800

Applicable component levels

  • R60A PSY

       UP

  • R60H PSY

       UP

  • R60I PSY

       UP

  • R60P PSY

       UP

  • R60S PSY

       UP

  • R60W PSY

       UP

  • R60Z PSY

       UP

  • R61A PSY

       UP

  • R61H PSY

       UP

  • R61I PSY

       UP

  • R61P PSY

       UP

  • R61S PSY

       UP

  • R61W PSY

       UP

  • R61Z PSY

       UP

  • R700 PSY

       UP

[{"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Product":{"code":"SSEQTP","label":"WebSphere Application Server"},"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"6.0","Line of Business":{"code":"LOB45","label":"Automation"}}]

Document Information

Modified date:
24 October 2021