Direct links to fixes
7.0.0-WS-WCServer-FP009
WebSphere Commerce Version 7.0.0.4 Fix Pack
WebSphere Commerce Version 7.0.0.3 Fix Pack
WebSphere Commerce Version 7.0.0.5 Fix Pack
WebSphere Commerce Version 7.0.0.6 Fix Pack
WebSphere Commerce Version 7.0.0.7 Fix Pack
WebSphere Commerce Version 7.0.0.8 Fix Pack
WebSphere Commerce Version 7.0.0.9 Fix Pack
APAR status
Closed as program error.
Error description
In IBM WebSphere Commerce, after the ActivityCleanUp job runs, explicit DynaCache invalidations are issued for each deleted activity record. This could result in thousands of invalidations, causing a large number of database locks. An analysis of javacores taken shows a large number of WebContainers are "waiting to enter": 3LKMONOBJECT com/ibm/ws/cache/Cache@0x71D421C8/0x71D421D4: owner "WebContainer : 5" (0x40A5BA00), entry count 2 3LKWAITERQ Waiting to enter: 3LKWAITER "Default : DMN2" (0x379FA700) 3LKWAITER "WebContainer : 0" (0x38664C00) 3LKWAITER "WebContainer : 1" (0x3EC89000) 3LKWAITER "WebContainer : 6" (0x41814D00) 3LKWAITER "WebContainer : 8" (0x361A3A00) 3LKWAITER "WebContainer : 14" (0x41FF5C00) 3LKWAITER "WebContainer : 15" (0x41FF6100) 3LKWAITER "WebContainer : 18" (0x41D02D00) 3LKWAITER "WebContainer : 20" (0x41CD4C00) 3LKWAITER "WebContainer : 21" (0x41CD5100) 3LKWAITER "WebContainer : 22" (0x4216E700) 3LKWAITER "WebContainer : 23" (0x4216EC00) 3LKWAITER "WebContainer : 24" (0x42EA2700) 3LKWAITER "WebContainer : 26" (0x43041B00) 3LKWAITER "WebContainer : 27" (0x43042000) 3LKWAITER "WebContainer : 30" (0x42EB3000) And the WebContainer causing the delay is performing DynaCache invalidations: 3XMTHREADINFO "WebContainer : 5" J9VMThread:0x40A5BA00, j9thread_t:0x40A576EC, java/lang/Thread:0x8ACD16B8, state:CW, prio=5 3XMTHREADINFO1 (native thread ID:0x1C7006F, native priority:0x5, native policy:UNKNOWN) 3XMTHREADINFO3 Java callstack: 4XESTACKTRACE at java/util/HashSet.contains(HashSet.java:138(Compiled Code)) 4XESTACKTRACE at com/ibm/ws/cache/HTODDependencyTable.containsCacheId(HTODDepende ncyTable.java:261(Compiled Code)) 4XESTACKTRACE at com/ibm/ws/cache/HTODDynacache.isCacheIdInAuxDepIdTable(HTODDyna cache.java:3433(Compiled Code)) 4XESTACKTRACE at com/ibm/ws/cache/CacheOnDisk.isCacheIdInAuxDepIdTable(CacheOnDis k.java(Compiled Code)) 4XESTACKTRACE at com/ibm/ws/cache/Cache.remove(Cache.java:1619(Compiled Code)) 4XESTACKTRACE at com/ibm/ws/cache/Cache.internalInvalidateById(Cache.java:1545(Co mpiled Code)) 4XESTACKTRACE at com/ibm/ws/cache/Cache.batchUpdate(Cache.java:1439(Compiled Code)) 4XESTACKTRACE at com/ibm/ws/cache/BatchUpdateDaemon.wakeUp(BatchUpdateDaemon.java :353(Compiled Code)) 4XESTACKTRACE at com/ibm/ws/cache/BatchUpdateDaemon.invalidateById(BatchUpdateDae mon.java:158(Compiled Code)) 4XESTACKTRACE at com/ibm/ws/cache/BatchUpdateDaemon.invalidateById(BatchUpdateDae mon.java:137(Compiled Code)) 4XESTACKTRACE at com/ibm/ws/cache/Cache.invalidateById(Cache.java:1379(Compiled Code)) 4XESTACKTRACE at com/ibm/ws/cache/DCacheBase.invalidateById(DCacheBase.java(Compi led Code)) 4XESTACKTRACE at com/ibm/ws/cache/DistributedObjectCacheAdapter.common_invalidate (DistributedObjectCacheAdapter.java:1111(Compiled Code)) 4XESTACKTRACE at com/ibm/ws/cache/DistributedMapImpl.invalidate(DistributedMapImp l.java(Compiled Code)) 4XESTACKTRACE at com/ibm/commerce/datatype/CacheRule$MyInvalidationIds.myIssueDyn amicCacheInvalidations(CacheRule.java:12102) 4XESTACKTRACE at com/ibm/commerce/datatype/CacheRule$MyInvalidationIds.myIssueDyn amicCacheInvalidations(CacheRule.java:12057(Compiled Code)) 4XESTACKTRACE at com/ibm/commerce/datatype/CacheRule$MyInvalidationIds.transactio nNotify(CacheRule.java:12124(Compiled Code)) 4XESTACKTRACE at com/ibm/commerce/datatype/CustomProperties.transactionNotify(Cus tomProperties.java:419(Compiled Code)) 4XESTACKTRACE at com/ibm/commerce/datatype/TransactionCache.transactionNotify(Tra nsactionCache.java:164(Compiled Code)) 4XESTACKTRACE at com/ibm/commerce/server/TransactionManager.commit(TransactionMan ager.java:138(Compiled Code)) ...
Local fix
As a workaround to this issue, WebSphere Commerce users can disable the ActivityCleanUp job and use the dbclean utility, or manually clean the activity data tables. The Out-of-Box ActivityCleanUp job parameter is: memberType=GRAS&status=TE&maxIdleMinutes=1440&maxRetainMinutes=1 440 The delete statement should be similar to the following: delete from ctxmgmt where status in ('T','E') and timestamp(lastaccesstime) < current timestamp - 1440 minutes and caller_id in (select users_id from users where registertype in ('G','R','A','S')); The status, retain/idle time and registertype values may be changed to meet your requirements. Additional information can be found here: Maintaining the Business Context tables using DBClean http://www-01.ibm.com/support/docview.wss?uid=swg21378083
Problem summary
USERS AFFECTED: WebSphere Commerce users on v6.0 and v7.0 who use DynaCache and the ActivityCleanUp job is enabled. PROBLEM ABSTRACT: WebSphere Commerce performance degradation due to large number of database locks caused by DynaCache invalidations BUSINESS IMPACT: Web site performance can be impacted due to database locks caused by a large number of DynaCache invalidations. RECOMMENDATION:
Problem conclusion
Downloading and installing the interim fix for this APAR resolves the issue. The interim fix changes the ActivityCleanUp job to submit a single DynaCache invalidation for all of the records deleted, rather than explicit invalidations for each record deleted. ------------------------------------------------------------- The latest available maintenance information can be obtained from the Recommended Fixes for WebSphere Commerce technote: http://www.ibm.com/support/docview.wss?rs=3046&uid=swg21261296
Temporary fix
Comments
APAR Information
APAR number
JR38217
Reported component name
WC BUS EDITION
Reported component ID
5724I3800
Reported release
700
Status
CLOSED PER
PE
NoPE
HIPER
NoHIPER
Special Attention
NoSpecatt
Submitted date
2010-11-09
Closed date
2010-12-24
Last modified date
2011-02-10
APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:
Fix information
Fixed component name
WC BUS EDITION
Fixed component ID
5724I3800
Applicable component levels
R700 PSY
UP
[{"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSYSYL","label":"WebSphere Commerce Enterprise"},"Component":"","ARM Category":[],"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"7.0","Edition":"","Line of Business":{"code":"","label":""}}]
Document Information
Modified date:
10 February 2011