Fixes are available
APAR status
Closed as program error.
Error description
--------------------------------------------- Users Affected: Users running ITCAM for WAS, J2EE and ITCAM for AD 7.1 DCs --------------------------------------------- Problem Description: In very rare situations, a deadlock may occur in ITCAM Data Collector between GC parser and TEMA connection threads during TEMA disconnect. This happens when the timing is just perfect for two nested synchronization blocks to be executed at the right time by two threads. One thread is reading the verbose GC log and another is shutting down the TEMA listener due to the connection between DC and TEMA was disconnected for some reason. The fix will corrects the order in which the locks are obtained so that A->B and B->A situation does not arise. The following stack can be found in the javacore: > 1LKDEADLOCK Deadlock detected !!! > NULL --------------------- > NULL > 2LKDEADLOCKTHR Thread "Thread-27" (0x0000010021B96000) > 3LKDEADLOCKWTR is waiting for: > 4LKDEADLOCKMON sys_mon_t:0x000001001E30A5A8 infl_mon_t: 0x000001001E30A5E8: > 4LKDEADLOCKOBJ com/ibm/tivoli/itcam/toolkit/ai/gccollector/GCProcessor@0x070000 00123FC138/0x07000000123FC150: > 3LKDEADLOCKOWN which is owned by: > 2LKDEADLOCKTHR Thread "Thread-29" (0x0000010021B4B900) > 3LKDEADLOCKWTR which is waiting for: > 4LKDEADLOCKMON sys_mon_t:0x000001001E30A520 infl_mon_t: 0x000001001E30A560: > 4LKDEADLOCKOBJ com/ibm/tivoli/itcam/toolkit/ai/gccollector/GCCollector@0x070000 001229A938/0x070000001229A950: > 3LKDEADLOCKOWN which is owned by: > 2LKDEADLOCKTHR Thread "Thread-27" (0x0000010021B96000) > > > > 3XMTHREADINFO "Thread-27" J9VMThread:0x0000010021B96000, j9thread_t:0x000001001E32DE20, java/lang/Thread:0x07000000123FD478, state:B, prio=5 > 3XMTHREADINFO1 (native thread ID:0x3E00023, native priority:0x5, native policy:UNKNOWN) > 3XMTHREADINFO3 Java callstack: > 4XESTACKTRACE at com/ibm/tivoli/itcam/toolkit/ai/gccollector/GCCollector.stopColl ecting(GCCollector.java:96) > 4XESTACKTRACE at com/ibm/tivoli/itcam/tema/gc/TEMAGCCollector.stopCollecting(TEMA GCCollector.java:397) > 4XESTACKTRACE at com/ibm/tivoli/itcam/tema/gc/TEMAGCCollector.handleNotification( TEMAGCCollector.java:369) > 4XESTACKTRACE at com/sun/jmx/interceptor/DefaultMBeanServerInterceptor$ListenerWr apper.handleNotification(DefaultMBeanServerInterceptor.java:1732 (Compiled Code)) > 4XESTACKTRACE at javax/management/NotificationBroadcasterSupport.handleNotificati on(NotificationBroadcasterSupport.java:257) > 4XESTACKTRACE at javax/management/NotificationBroadcasterSupport$SendNotifJob.run (NotificationBroadcasterSupport.java:322) > 4XESTACKTRACE at javax/management/NotificationBroadcasterSupport$1.execute(Notifi cationBroadcasterSupport.java:307) > 4XESTACKTRACE at javax/management/NotificationBroadcasterSupport.sendNotification (NotificationBroadcasterSupport.java:229(Compiled Code)) > 4XESTACKTRACE at com/ibm/tivoli/itcam/tema/jmxconnector/server/JMXConnectorServer .setCollectionEnabled(JMXConnectorServer.java:350) > 4XESTACKTRACE at com/ibm/tivoli/itcam/tema/jmxconnector/server/JMXConnectorServer .connectionClosed(JMXConnectorServer.java:680) > 4XESTACKTRACE at com/ibm/tivoli/itcam/tema/jmxconnector/server/JMXConnectorServer .access$500(JMXConnectorServer.java:45) > 4XESTACKTRACE at com/ibm/tivoli/itcam/tema/jmxconnector/server/JMXConnectorServer $ConnectionEstablisher.run(JMXConnectorServer.java:1288) > 3XMTHREADINFO3 Native callstack: > 4XENATIVESTACK _event_wait+0x2b8 (0x09000000004DBD5C [libpthreads.a+0x16d5c]) > 4XENATIVESTACK _cond_wait_local+0x4e4 (0x09000000004E9A68 [libpthreads.a+0x24a68]) > 4XENATIVESTACK _cond_wait+0xbc (0x09000000004EA040 [libpthreads.a+0x25040]) > 4XENATIVESTACK pthread_cond_wait+0x1a8 (0x09000000004EACAC [libpthreads.a+0x25cac]) > 4XENATIVESTACK (0x09000000031DF17C [libj9thr24.so+0x417c]) > 4XENATIVESTACK (0x09000000031DEF40 [libj9thr24.so+0x3f40]) > 4XENATIVESTACK (0x09000000031DEEE0 [libj9thr24.so+0x3ee0]) > 4XENATIVESTACK (0x0900000003161A34 [libj9vm24.so+0x10a34]) > 4XENATIVESTACK (0x0900000003160FF4 [libj9vm24.so+0xfff4]) > 4XENATIVESTACK (0x0900000003196FE8 [libj9vm24.so+0x45fe8]) > 4XENATIVESTACK (0x0900000003196748 [libj9vm24.so+0x45748]) > 4XENATIVESTACK (0x09000000031BDEE4 [libj9vm24.so+0x6cee4]) > 4XENATIVESTACK (0x090000000315CC20 [libj9vm24.so+0xbc20]) > 4XENATIVESTACK (0x09000000031F0F94 [libj9prt24.so+0x1f94]) > 4XENATIVESTACK (0x090000000315CB40 [libj9vm24.so+0xbb40]) > 4XENATIVESTACK (0x09000000031DCC70 [libj9thr24.so+0x1c70]) > 4XENATIVESTACK _pthread_body+0xf0 (0x09000000004C8D54 [libpthreads.a+0x3d54])
Local fix
Problem summary
**************************************************************** USERS AFFECTED: Users running ITCAM for WAS, J2EE and ITCAM for AD 7.1 DCs **************************************************************** PROBLEM SUMMARY: In very rare situations, a deadlock may occur in ITCAM Data Collector between GC parser and TEMA connection threads during TEMA disconnect. This happens when the timing is just perfect for two nested synchronization blocks to be executed at the right time by two threads. One thread is reading the verbose GC log and another is shutting down the TEMA listener due to the connection between DC and TEMA was disconnected for some reason. The fix will corrects the order in which the locks are obtained so that A->B and B->A situation does not arise. The following stack can be found in the javacore: > 1LKDEADLOCK Deadlock detected !!! > NULL --------------------- > NULL > 2LKDEADLOCKTHR Thread "Thread-27" (0x0000010021B96000) > 3LKDEADLOCKWTR is waiting for: > 4LKDEADLOCKMON sys_mon_t:0x000001001E30A5A8 infl_mon_t: 0x000001001E30A5E8: > 4LKDEADLOCKOBJ com/ibm/tivoli/itcam/toolkit/ai/gccollector/GCProcessor@0x070000 00123FC138/0x07000000123FC150: > 3LKDEADLOCKOWN which is owned by: > 2LKDEADLOCKTHR Thread "Thread-29" (0x0000010021B4B900) > 3LKDEADLOCKWTR which is waiting for: > 4LKDEADLOCKMON sys_mon_t:0x000001001E30A520 infl_mon_t: 0x000001001E30A560: > 4LKDEADLOCKOBJ com/ibm/tivoli/itcam/toolkit/ai/gccollector/GCCollector@0x070000 001229A938/0x070000001229A950: > 3LKDEADLOCKOWN which is owned by: > 2LKDEADLOCKTHR Thread "Thread-27" (0x0000010021B96000) > > > > 3XMTHREADINFO "Thread-27" J9VMThread:0x0000010021B96000, j9thread_t:0x000001001E32DE20, java/lang/Thread:0x07000000123FD478, state:B, prio=5 > 3XMTHREADINFO1 (native thread ID:0x3E00023, native priority:0x5, native policy:UNKNOWN) > 3XMTHREADINFO3 Java callstack: > 4XESTACKTRACE at com/ibm/tivoli/itcam/toolkit/ai/gccollector/GCCollector.stopColl ecting(GCCollector.java:96) > 4XESTACKTRACE at com/ibm/tivoli/itcam/tema/gc/TEMAGCCollector.stopCollecting(TEMA GCCollector.java:397) > 4XESTACKTRACE at com/ibm/tivoli/itcam/tema/gc/TEMAGCCollector.handleNotification( TEMAGCCollector.java:369) > 4XESTACKTRACE at
Problem conclusion
The fix for this APAR is contained in the following maintenance releases: | interim fix | 7.1.0.3-TIV-ITCAMAD-IF0008 | interim fix | 6.1.0.6-TIV-ITCAMJ2EE_MP-IF0006
Temporary fix
Comments
APAR Information
APAR number
PM99072
Reported component name
TCAM AD WAS DC
Reported component ID
5724Y92DC
Reported release
710
Status
CLOSED PER
PE
NoPE
HIPER
NoHIPER
Special Attention
NoSpecatt
Submitted date
2013-10-15
Closed date
2013-11-08
Last modified date
2013-11-08
APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:
Fix information
Fixed component name
TCAM AD WAS DC
Fixed component ID
5724Y92DC
Applicable component levels
R710 PSY
UP
[{"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Product":{"code":"SS3PGL","label":"Tivoli Composite Application Manager for WebSphere"},"Component":"","ARM Category":[],"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"710","Edition":"","Line of Business":{"code":"LOB45","label":"Automation"}}]
Document Information
Modified date:
27 July 2021