APAR status
Closed as program error.
Error description
Excessive TTL synchronization work triggers hung thread messages A WXS client using near cache invalidation and the TTL evictor can have hung threads when a lot of invalidation work is needed. [6/14/13 4:00:43:011 EDT] 0000006f XSThreadPool W CWOBJ7853W: Detected a hung thread named "XIOPrimaryPool : 4" TID:a1 BLOCKED. Executing since Fri Jun 14 04:00:15 EDT 2013. Stack Trace: com.ibm.ws.objectgrid.OGEventListenerHandler.fireTransactionEnd (OGEventListenerHandler.java:408) com.ibm.ws.objectgrid.SessionImpl.afterCompletion(SessionImpl.j ava:2459) com.ibm.ws.objectgrid.SessionImpl.commit(SessionImpl.java:2214) com.ibm.ws.xs.cacheinvalidator.client.NearCacheSynchronizer.inv alidate(NearCacheSynchronizer.java:168) com.ibm.ws.xs.cacheinvalidator.client.NearCacheSubscriber.onMes sage(NearCacheSubscriber.java:114) com.ibm.ws.xs.pubsub.subscription.Subscriber.processMessage(Sub scriber.java:358) com.ibm.ws.xs.pubsub.subscription.Subscriber.receiveTopicMessag e(Subscriber.java:303) com.ibm.ws.xs.pubsub.subscription.Subscriber.receive(Subscriber .java:186) Depending on how long the threads hang the client may detect a disconnect from the server and proactively disable the near cache: CWOBJ7663W: Near-cache invalidation notification has been disconnected for grid ...the near cache has been disabled
Local fix
Problem summary
**************************************************************** * USERS AFFECTED: Usesrs that have near cache time-to-live * * (TTL) * * synchronization enabled. * **************************************************************** * PROBLEM DESCRIPTION: High levels of client activity can * * trigger excessive TTL synchronization * * notification messages. These messages * * block transactions, resulting in a * * bottleneck. Threads seem to be hung. * **************************************************************** * RECOMMENDATION: * **************************************************************** Large numbers of hung thread messages are written in the client process logs. The messages indicate that the threads are blocked on IBM eXtremeIO (XIO) activity or EventListener activity: [6/14/13 4:00:43:006 EDT] 0000006f XSThreadPool W CWOBJ7853W: Detected a hung thread named "XIOPrimaryPool : 0" TID:74 BLOCKED. Executing since Fri Jun 14 04:00:15 EDT 2013. Stack Trace: com.ibm.ws.xsspi.xio.actor.XIOReferable.dispatch(XIORefe rable.j ava:114) com.ibm.ws.xsspi.xio.actor.XIORegistry.sendToTarget(XIOR egistry .java:824) com.ibm.ws.xs.xio.transport.channel.XIORegistryRunnable. run(XIO RegistryRunnable.java:48) java.util.concurrent.ThreadPoolExecutor$Worker.runTask(T hreadPo olExecutor.java:897) java.util.concurrent.ThreadPoolExecutor$Worker.run(Threa dPoolEx ecutor.java:919) com.ibm.ws.objectgrid.thread.XSThreadPool$Worker.run(XST hreadPo ol.java:280) [6/14/13 4:00:43:010 EDT] 0000006f XSThreadPool W CWOBJ7853W: Detected a hung thread named "XIOPrimaryPool : 1" TID:9c BLOCKED. Executing since Fri Jun 14 04:00:15 EDT 2013. Stack Trace: com.ibm.ws.objectgrid.OGEventListenerHandler.fireTransac tionEnd (OGEventListenerHandler.java:408) com.ibm.ws.objectgrid.SessionImpl.afterCompletion(Sessio nImpl.j ava:2459) com.ibm.ws.objectgrid.SessionImpl.commit(SessionImpl.jav a:2214) com.ibm.ws.xs.cacheinvalidator.client.NearCacheSynchroni zer.inv alidate(NearCacheSynchronizer.java:168) com.ibm.ws.xs.cacheinvalidator.client.NearCacheSubscribe r.onMes sage(NearCacheSubscriber.java:114) com.ibm.ws.xs.pubsub.subscription.Subscriber.processMess age(Sub scriber.java:358) com.ibm.ws.xs.pubsub.subscription.Subscriber.receiveTopi cMessag e(Subscriber.java:303) com.ibm.ws.xs.pubsub.subscription.Subscriber.receive(Sub scriber .java:186) com.ibm.ws.xsspi.xio.actor.XIOReferable.dispatch(XIORefe rable.j ava:114) com.ibm.ws.xsspi.xio.actor.XIORegistry.sendToTarget(XIOR egistry .java:824) com.ibm.ws.xs.xio.transport.channel.XIORegistryRunnable. run(XIO RegistryRunnable.java:48) java.util.concurrent.ThreadPoolExecutor$Worker.runTask(T hreadPo olExecutor.java:897) java.util.concurrent.ThreadPoolExecutor$Worker.run(Threa dPoolEx ecutor.java:919) com.ibm.ws.objectgrid.thread.XSThreadPool$Worker.run(XST hreadPo ol.java:280)
Problem conclusion
The TTL synchronization messages are sent asynchronously until the client transaction completes. As a result, transactions are not blocked during the completion of the TTL synchronization flow.
Temporary fix
Comments
APAR Information
APAR number
PM91216
Reported component name
WS EXTREME SCAL
Reported component ID
5724X6702
Reported release
860
Status
CLOSED PER
PE
NoPE
HIPER
NoHIPER
Special Attention
NoSpecatt
Submitted date
2013-06-17
Closed date
2013-06-21
Last modified date
2013-06-21
APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:
Fix information
Fixed component name
WS EXTREME SCAL
Fixed component ID
5724X6702
Applicable component levels
R860 PSY
UP
[{"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Product":{"code":"SSTVLU","label":"WebSphere eXtreme Scale"},"Component":"","ARM Category":[],"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"860","Edition":"","Line of Business":{"code":"LOB45","label":"Automation"}}]
Document Information
Modified date:
21 June 2013