PI06011: An eXtreme Scale deadlock can occur when an ObjectGridEventListener end method creates a transaction that

APAR status

• Closed as program error.

Error description

• A deadlock can occur when an ObjectGridEventListener
  transaction end method creates a transaction that times out.
  The following example resembles the time out errors that you
  can experience:
  
  Deadlock detected !!!
  
  
  Thread "WXS : 896: TTLEvictorForMap"
  is waiting for:
  com/ibm/ws/objectgrid/SessionImpl
  which is owned by:
  Thread "TransactionManager : 9"
  which is waiting for:
  Thread "WXS : 896: TTLEvictorForMap-Session
  
  
  "WXS : 896: TTLEvictorForMap-Session"
  Blocked on: com/ibm/ws/objectgrid/SessionImpl
  Owned
  by: "TransactionManager : 9"
  Java callstack:
  at
  com/ibm/ws/objectgrid/ObjectMapImpl.checkIfPermittedDuringCommit
  (ObjectMapImpl.java:1733(Compiled Code))
  at
  com/ibm/ws/objectgrid/ObjectMapImpl.insert(ObjectMapImpl.java:13
  05(Compiled Code))
  at
  com/ibm/ws/objectgrid/ObjectMapImpl.insert(ObjectMapImpl.java:12
  89(Compiled Code))
  at
  com/mypackage/EvictionEventPublisher.notifyEviction(Evictio
  nEventPublisher.java:122(Compiled Code))
  at
  com/mypackage/EvictionEventPublisher.transactionEnd(Evictio
  nEventPublisher.java:93(Compiled Code))
  at
  com/ibm/ws/objectgrid/OGEventListenerHandler.fireTransactionEnd(
  OGEventListenerHandler.java:461(Compiled Code))
  (entered lock: [B@0x000000004314C600, entry count: 1)
  at
  com/ibm/ws/objectgrid/SessionImpl.afterCompletion(SessionImpl.ja
  va:2275(Compiled Code))
  at
  com/ibm/ws/objectgrid/SessionImpl.commit(SessionImpl.java:2036(C
  ompiledCode))
  (entered lock:
  com/ibm/ws/objectgrid/SessionImpl@0x000000004520E608,
  entry count: 1)
  at
  com/ibm/ws/objectgrid/map/BaseMap.evictMapEntries(BaseMap.java:3
  707(Compiled Code))
  at
  com/ibm/ws/objectgrid/plugins/TTLEvictor.run(TTLEvictor.java:153
  5(Compiled Code))
  at
  com/ibm/ws/objectgrid/util/security/SecurityContextRunnable$2.ru
  n(SecurityContextRunnable.java:111(Compiled Code))
  at
  com/ibm/ws/security/auth/ContextManagerImpl.runAs(ContextManager
  Impl.java:5453(Compiled Code))
  at
  com/ibm/ws/security/auth/ContextManagerImpl.runAsSystem(ContextM
  anagerImpl.java:5579(Compiled Code))
  at
  com/ibm/ws/objectgrid/util/security/SecurityContextRunnable.runW
  ithServerContext(SecurityContextRunnable.java:109(Compiled
  Code))
  at
  com/ibm/ws/objectgrid/util/security/SecurityContextRunnable.run(
  SecurityContextRunnable.java:68(Compiled Code))
  at
  java/util/concurrent/ThreadPoolExecutor$Worker.runTask(ThreadPoo
  lExecutor.java:897(Compiled Code))
  at
  java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExe
  cutor.java:919(Compiled Code))
  at java/lang/Thread.run(Thread.java:772(Compiled Code))
  
  "TransactionManager : 9"
  Blocked on: [B@0x000000004314C600 Owned by: "WXS : 896:
  TTLEvictorForMap-Session"
  Java callstack:
  at
  com/ibm/ws/objectgrid/OGEventListenerHandler.fireTransactionEnd(
  OGEventListenerHandler.java:406(Compiled Code))
  at
  com/ibm/ws/objectgrid/SessionImpl.afterCompletion(SessionImpl.ja
  va:2275(Compiled Code))
  at
  com/ibm/ws/objectgrid/SessionImpl.makeCleanForRollbackOnly(Sessi
  onImpl.java:2659)
  (entered lock:
  com/ibm/ws/objectgrid/SessionImpl@0x00000000451F0E40,
  entry count: 1)
  at
  com/ibm/ws/objectgrid/transaction/TransactionManager.processAsyn
  cRollback(TransactionManager.java:468)
  at
  com/ibm/ws/objectgrid/transaction/TransactionManager.rollbackNex
  tBucket(TransactionManager.java:363(Compiled Code))
  at
  com/ibm/ws/objectgrid/transaction/TransactionManager$TimeoutReap
  er$1.run(TransactionManager.java:723(Compiled Code))
  at
  java/util/concurrent/ThreadPoolExecutor$Worker.runTask(ThreadPoo
  lExecutor.java:897(Compiled Code))
  at
  java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExe
  cutor.java:919(Compiled Code))
  at java/lang/Thread.run(Thread.java:772(Compiled Code))
  

Local fix

Problem summary

• ****************************************************************
  * USERS AFFECTED:  WebSphere eXtreme Scale users who start     *
  *                  transactions within their                   *
  *                  ObjectGridEventListener transactionEnd().   *
  ****************************************************************
  * PROBLEM DESCRIPTION: The ObjectGridEventListener             *
  *                      transactionEnd() times out and causes   *
  *                      a deadlock between threads.             *
  ****************************************************************
  * RECOMMENDATION:                                              *
  ****************************************************************
  A deadlock occurs between the ObjectGridEventListener
  transactionEnd() thread and a thread that cleans up the
  timed out transaction.
  The thread calling the transaction end method has a listener
  lock. If a user starts a new transaction in the transaction end
  method that times out, the reaper thread obtains a lock
  on the session and waits to get the listener lock. Meanwhile,
  the first thread can wait to get a session lock and a deadlock
  occurs between these two threads.
  

Problem conclusion

• The code running in the reaper thread was changed to release
  the session lock before attempting to get the listener lock,
  which prevents a deadlock from occurring.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  WS EXTREME SCAL

• Fixed component ID

  5724X6702

Applicable component levels

• R850 PSY

     UP

• R860 PSY

     UP