Fixes are available
PM11713: Out Of Memory exceptions after large spikes in load with large tran..
7.0.0.11: WebSphere Application Server V7.0 Fix Pack 11 for IBM i
7.0.0.11: WebSphere Application Server V7.0 Fix Pack 11 for Windows
7.0.0.11: WebSphere Application Server V7.0 Fix Pack 11 for HP-UX
7.0.0.11: WebSphere Application Server V7.0 Fix Pack 11 for AIX
7.0.0.11: Java SDK 1.6 SR7 Cumulative Fix for WebSphere Application Server
7.0.0.11: WebSphere Application Server V7.0 Fix Pack 11 for Solaris
7.0.0.11: WebSphere Application Server V7.0 Fix Pack 11 for Linux
6.1.0.33: Java SDK 1.5 SR12 FP1 Cumulative Fix for WebSphere
7.0.0.13: WebSphere Application Server V7.0 Fix Pack 13 for AIX
7.0.0.13: WebSphere Application Server V7.0 Fix Pack 13 for HP-UX
7.0.0.13: WebSphere Application Server V7.0 Fix Pack 13 for IBM i
7.0.0.13: WebSphere Application Server V7.0 Fix Pack 13 for Linux
7.0.0.13: WebSphere Application Server V7.0 Fix Pack 13 for Solaris
7.0.0.13: WebSphere Application Server V7.0 Fix Pack 13 for Windows
7.0.0.13: Java SDK 1.6 SR8FP1 Cumulative Fix for WebSphere Application Server
6.1.0.35: Java SDK 1.5 SR12 FP2 Cumulative Fix for WebSphere
7.0.0.15: WebSphere Application Server V7.0 Fix Pack 15 for AIX
7.0.0.15: Java SDK 1.6 SR9 Cumulative Fix for WebSphere Application Server
7.0.0.15: WebSphere Application Server V7.0 Fix Pack 15 for HP-UX
7.0.0.15: WebSphere Application Server V7.0 Fix Pack 15 for IBM i
7.0.0.15: WebSphere Application Server V7.0 Fix Pack 15 for Linux
7.0.0.15: WebSphere Application Server V7.0 Fix Pack 15 for Solaris
7.0.0.15: WebSphere Application Server V7.0 Fix Pack 15 for Windows
6.1.0.37: Java SDK 1.5 SR12 FP3 Cumulative Fix for WebSphere
7.0.0.17: WebSphere Application Server V7.0 Fix Pack 17
7.0.0.17: Java SDK 1.6 SR9 FP1 Cumulative Fix for WebSphere Application Server
7.0.0.19: WebSphere Application Server V7.0 Fix Pack 19
7.0.0.21: WebSphere Application Server V7.0 Fix Pack 21
7.0.0.23: WebSphere Application Server V7.0 Fix Pack 23
7.0.0.25: WebSphere Application Server V7.0 Fix Pack 25
7.0.0.27: WebSphere Application Server V7.0 Fix Pack 27
7.0.0.29: WebSphere Application Server V7.0 Fix Pack 29
6.1.0.47: WebSphere Application Server V6.1 Fix Pack 47
7.0.0.31: WebSphere Application Server V7.0 Fix Pack 31
7.0.0.27: Java SDK 1.6 SR13 FP2 Cumulative Fix for WebSphere Application Server
7.0.0.33: WebSphere Application Server V7.0 Fix Pack 33
7.0.0.35: WebSphere Application Server V7.0 Fix Pack 35
7.0.0.37: WebSphere Application Server V7.0 Fix Pack 37
7.0.0.39: WebSphere Application Server V7.0 Fix Pack 39
7.0.0.41: WebSphere Application Server V7.0 Fix Pack 41
7.0.0.43: WebSphere Application Server V7.0 Fix Pack 43
7.0.0.45: WebSphere Application Server V7.0 Fix Pack 45
6.1.0.39: Java SDK 1.5 SR12 FP4 Cumulative Fix for WebSphere Application Server
6.1.0.41: Java SDK 1.5 SR12 FP5 Cumulative Fix for WebSphere Application Server
6.1.0.43: Java SDK 1.5 SR13 Cumulative Fix for WebSphere Application Server
6.1.0.45: Java SDK 1.5 SR14 Cumulative Fix for WebSphere Application Server
6.1.0.47: Java SDK 1.5 SR16 Cumulative Fix for WebSphere Application Server
7.0.0.19: Java SDK 1.6 SR9 FP2 Cumulative Fix for WebSphere Application Server
7.0.0.21: Java SDK 1.6 SR9 FP2 Cumulative Fix for WebSphere
7.0.0.23: Java SDK 1.6 SR10 FP1 Cumulative Fix for WebSphere
7.0.0.25: Java SDK 1.6 SR11 Cumulative Fix for WebSphere Application Server
7.0.0.27: Java SDK 1.6 SR12 Cumulative Fix for WebSphere Application Server
7.0.0.29: Java SDK 1.6 SR13 FP2 Cumulative Fix for WebSphere Application Server
7.0.0.45: Java SDK 1.6 SR16 FP60 Cumulative Fix for WebSphere Application Server
7.0.0.31: Java SDK 1.6 SR15 Cumulative Fix for WebSphere Application Server
7.0.0.35: Java SDK 1.6 SR16 FP1 Cumulative Fix for WebSphere Application Server
7.0.0.37: Java SDK 1.6 SR16 FP3 Cumulative Fix for WebSphere Application Server
7.0.0.39: Java SDK 1.6 SR16 FP7 Cumulative Fix for WebSphere Application Server
7.0.0.41: Java SDK 1.6 SR16 FP20 Cumulative Fix for WebSphere Application Server
7.0.0.43: Java SDK 1.6 SR16 FP41 Cumulative Fix for WebSphere Application Server
APAR status
Closed as program error.
Error description
The primary symptom is an OutOfMemoryError. This may cause CNTR0019E or CNTR0020E errors, since the memory that is not being cleaned up is EJB cache memory. . The heapdump will resemble the following. com/ibm/ejs/util/cache/BackgroundLruEvictionStrategy, which has a com/ibm/ejs/util/cache/CacheElementEnumerator, which has an array of com/ibm/ejs/util/cache/Element objects. Each com/ibm/ejs/util/cache/Element has a com/ibm/ejs/container/activator/TransactionKey, which has a com/ibm/ejs/container/ContainerTx, which then has an ArrayList which contains thousands of com/ibm/ejs/container/ContainerManagedBeanO objects. . The problem was reported using IBM WebSphere Commerce Suite, so the beans in the EJB cache are EJBs from that product. The issue is not with the CommerceSuite beans, but with how the WebSphere Application Server cache is handling them. . The key symptom in terms of the heap is the Array of thousands of ContainerManagedBeanO objects tied to a CacheElementEnumerator. .-------------------------------------------------------- This shows that the memory that is potentially being leaked is tied to the EJB cache that WebSphere AppServer maintains. 317,803,376 (33%) [176] 5 com/ibm/ejs/util/cache/BackgroundLruEvictionStrategy 0x72ebe0c8 317,803,064 (33%) [56] 3 com/ibm/ejs/util/cache/CacheElementEnumerator 0x8243f630 317,802,912 (33%) [472] 26 array of java/lang/Object 0xa70484b8 .-------------------------------------- Here is one of the elements in the EJB Cache. 238,307,992 (25%) [40] 2 com/ibm/ejs/util/cache/Element 0xb6ae1778 238,307,256 (25%) [24] 2 com/ibm/ejs/container/activator/TransactionKey 0xb6ae17a0 238,306,896 (25%) [136] 10 com/ibm/ejs/container/ContainerTx 0x88a960a0 164,283,672 (17%) [24] 1 java/util/ArrayList 0x88d8f5c8 . Here is that giant array of beans. It is directly tied to a cached EJB. The array consists of lots and lots of com/ibm/commerce/order/objects/EJSCMPTradingPositionContainerXML DefinHomeBean_c498873a beans. What this says to me is that the order objects are not being cleaned up and are accumulating. There is an array of 185,387 ContainerManagedBeanO objects. 164,283,648 (17%) [794,328] 185,387 array of java/lang/Object 0xa4bd5008 21,504 (0%) [176] 10 com/ibm/ejs/container/ContainerManagedBeanO 0x761eb0c8 21,504 (0%) [176] 10 com/ibm/ejs/container/ContainerManagedBeanO 0x96b5b6e0 21,504 (0%) [176] 10 com/ibm/ejs/container/ContainerManagedBeanO 0x9a4ea550 21,224 (0%) [48] 4 com/ibm/commerce/order/objects/TradingPositionContainerXMLDefini tionBean 0x9a4ea600 104 (0%) [24] 1 com/ibm/ejs/j2c/HandleList 0x9a4ea538 825,919,352 (88%) [200] 33 com/ibm/ejs/container/EJSContainer 0x8240da48 959,128 (0%) [160] 18 com/ibm/commerce/order/objects/EJSCMPTradingPositionContainerXML DefinHom eBean_c498873a 0x8f0e5d38 957,072 (0%) [56] 4 com/ibm/ejs/csi/PoolImplThreadSafe 0x8f0e8070 1,576 (0%) [248] 15 com/ibm/ws/pmi/server/modules/BeanModule 0x8f0e59c0 872 (0%) [80] 16 array of java/lang/String 0x7a456348 128 (0%) [72] 5 com/ibm/commerce/order/objects/EJSJDBCPersisterCMPTradingPositio nContainerXMLDefinitionBean_c498873a 0x8f0c2948 24 (0%) [24] 2 com/ibm/ejs/container/activator/OptCEntityActivationStrategy 0x82495070 16 (0%) [16] 0 com/ibm/ejs/container/lock/LockStrategy 0x9326ee08 21,504 (0%) [176] 10 com/ibm/ejs/container/ContainerManagedBeanO 0x8f0fe698 21,224 (0%) [48] 4 com/ibm/commerce/order/objects/TradingPositionContainerXMLDefini tionBean 0x8f0fe748 104 (0%) [24] 1 com/ibm/ejs/j2c/HandleList 0x8f0fe680 825,919,352 (88%) [200] 33 com/ibm/ejs/container/EJSContainer 0x8240da48 959,128 (0%) [160] 18 com/ibm/commerce/order/objects/EJSCMPTradingPositionContainerXML DefinHomeBean_c498873a 0x8f0e5d38
Local fix
Problem summary
**************************************************************** * USERS AFFECTED: All users of IBM WebSphere Application * * Server containing large or long-running * * transactions. * **************************************************************** * PROBLEM DESCRIPTION: Users can experience Out Of Memory * * exceptions following large spikes in * * load with large transactions. * **************************************************************** * RECOMMENDATION: * **************************************************************** Out Of Memory issues may result after large spikes in load. Analysis of the heap dumps should expose large "inactive" transactions with EJB references being held by the BackgroundLruEvictionStrategy. This APAR is a follow-on APAR to PK98076 that addresses a separate issue in the same area of code.
Problem conclusion
Modified the EJBContainer code to completely clear the copy of the cache bucket so that all items will be garbage collected prior to the next valid sweep. The fix for this APAR is currently targeted for inclusion in fix packs 7.0.0.11, 6.1.0.33, Feature Pack for EJB3.0 6.1.0.33, and 6.0.2.43. Please refer to the Recommended Updates page for delivery information: http://www.ibm.com/support/docview.wss?rs=180&uid=swg27004980
Temporary fix
Comments
APAR Information
APAR number
PM11713
Reported component name
WEBSPHERE APP S
Reported component ID
5724J0800
Reported release
60A
Status
CLOSED PER
PE
NoPE
HIPER
NoHIPER
Special Attention
NoSpecatt
Submitted date
2010-04-07
Closed date
2010-04-28
Last modified date
2010-04-28
APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:
Fix information
Fixed component name
WEBSPHERE APP S
Fixed component ID
5724J0800
Applicable component levels
R60A PSY
UP
R60H PSY
UP
R60I PSY
UP
R60P PSY
UP
R60S PSY
UP
R60W PSY
UP
R60Z PSY
UP
R61A PSY
UP
R61H PSY
UP
R61I PSY
UP
R61P PSY
UP
R61S PSY
UP
R61W PSY
UP
R61Z PSY
UP
R700 PSY
UP
Document Information
Modified date:
24 October 2021