PM84487: APPLICATION THREADS HUNG DUE TO UNRELEASED ITCAM LOCK.

7.1.0.3-TIV-ITCAMAD-IF0008: ITCAM for Application Diagnostics V7.1 FP3 IF8
7.1.0-TIV-ITCAMAD-FP0004: ITCAM for Application Diagnostics V7.1 Fix Pack 4

APAR status

• Closed as program error.

Error description

• In very rare cases, a thread that has acquired one of ITCAM
  internal locks seems to be abruptly terminated (instead of
  gracefully terminating) while holding on to the lock.  This
  causes the lock not to be released and other threads waiting for
  the lock are hung.
  
  A thread dump (e.g. javacore from a IBM JVM) will indicate that
  application threads are waiting for a thread that has a stack
  trace as follows:
  
  at java/lang/Object.wait(Object.java:167(Compiled Code))
  at
  com/ibm/tivoli/itcam/toolkit/ai/lockmanagement/LockAcquisitionRe
  quest.waitForLock(LockAcquisitionRequest.java:174)
  at
  com/ibm/tivoli/itcam/toolkit/ai/lockmanagement/Lock.lockAcquire(
  Lock.java:1071(Compiled Code))
  at
  com/ibm/tivoli/itcam/toolkit/ai/lockmanagement/Lock.obtainExclus
  ive(Lock.java:443(Compiled Code))
  
  This thread is waiting to acquire a lock that is locked by a
  thread that is no longer present in the thread dump, which
  indicates that it has terminated.
  

Local fix

Problem summary

• ****************************************************************
  * PROBLEM DESCRIPTION:
  *
  * In very rare cases, an application thread holding an internal
  * ITCAM lock has seen to suddenly die without releasing the
  * lock.  In such cases, other application threads waiting for
  * the lock hang indefinitely.
  *
  * The application threads that are hung show stack trace
  * similar to the following:
  *
  * at java/lang/Object.wait(Object.java:167(Compiled Code))
  * at
  com/ibm/tivoli/itcam/toolkit/ai/lockmanagement/LockAcquisitionRe
  quest.waitForLock(LockAcquisitionRequest.java:174)
  * at
  com/ibm/tivoli/itcam/toolkit/ai/lockmanagement/Lock.lockAcquire(
  Lock.java:1071(Compiled Code))
  * at
  com/ibm/tivoli/itcam/toolkit/ai/lockmanagement/Lock.obtainExclus
  ive(Lock.java:443(Compiled Code))
  *
  ****************************************************************
  

Problem conclusion

• This APAR introduces a check mechanism where by threads waiting
  for a lock for more than 60 seconds will check if the thread
  holding the lock is still alive.  If alive, it will print out
  the stack trace of the holding threads.  If the thread holding
  the exclusive lock is no longer alive, the lock is released so
  that other waiting threads can proceed with their execution.
  
  Also, the following property has been added to control the
  amount of waiting time (in milliseconds):
  
  am.camtoolkit.gpe.runtime.exclusivelock.check.time
  
  If this property is set to zero, then no lock checking will be
  done.
  
  
  The fix for this APAR is contained in the following maintenance
  packages:
  
    | interim fix | 7.1.0.3_TIV_ITCAMAD_IF0008
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  TCAM AD WAS DC

• Fixed component ID

  5724Y92DC

Applicable component levels

• R710 PSY

     UP