IV67167: ITNM BOOK HANGS AT END OF LOADIDML PROCESS WITH A THREAD DEADLOCK.

APAR status

• Closed as program error.

Error description

• When running loadidml, the ITNM book did not complete
  processing. All objects were loaded, but loadidml is still
  running.  In the logs you can see that the processing completed
  via the 'Objects left: 0' message added with IV63998;
  
  
  2014-11-19 07:39:21,069  [t93282]  INFO bulk.CdbRandomAccess -
  Cache
  Contents (0), implicit relationships:0, explcit relationships:0
  2014-11-19 07:39:21,069  [t93282]  INFO bulk.CdbRandomAccess -
  End Cache
  Contents
  
  2014-11-19 07:39:21,069  [t93282]  DEBUG bulk.CdbRandomAccess -
  Objects
  left: 0 ,Status: -1=0 ,0=0 ,1=0 2=0 ,statusLevel=2
  2014-11-19 07:39:21,069  [t93282]  INFO bulk.CdbBulkStatistics -
  Total
  time in core: 7066109 milliseconds
  
  
  However, when taking a thread dump of the storage service(or
  topology if domain mode) java process, you can see a deadlock in
  the javacore file;
  
  
  1LKDEADLOCK    Deadlock detected !!!
  NULL           ---------------------
  NULL
  2LKDEADLOCKTHR  Thread "RMI TCP Connection(8048)-10.119.10.166"
  (0x000000000148F700)
  3LKDEADLOCKWTR    is waiting for:
  4LKDEADLOCKMON      sys_mon_t:0x00002AAAD2635928 infl_mon_t:
  0x00002AAAD26359A0:
  4LKDEADLOCKOBJ
  com/ibm/cdb/topomgr/engine/TopologyStorageEngine@0x0000000699E3B
  6E8
  3LKDEADLOCKOWN    which is owned by:
  2LKDEADLOCKTHR  Thread "ChgWork-2" (0x00000000014C7A00)
  3LKDEADLOCKWTR    which is waiting for:
  4LKDEADLOCKMON      sys_mon_t:0x00002AAAD26379F8 infl_mon_t:
  0x00002AAAD2637A70:
  4LKDEADLOCKOBJ
  org/datanucleus/jdo/JDOPersistenceManager@0x00000006D054A528
  3LKDEADLOCKOWN    which is owned by:
  2LKDEADLOCKTHR  Thread "RMI TCP Connection(8048)-10.119.10.166"
  (0x000000000148F700)
  

Local fix

• NA
  

Problem summary

• When running loadidml, the ITNM book did not complete
  processing. All objects were loaded, but loadidml is still
  running.  In the logs you can see that the processing completed
  via the 'Objects left: 0' message added with IV63998;
  
  
  2014-11-19 07:39:21,069  ít93282ù  INFO bulk.CdbRandomAccess -
  Cache
  Contents (0), implicit relationships:0, explcit relationships:0
  2014-11-19 07:39:21,069  ít93282ù  INFO bulk.CdbRandomAccess -
  End Cache
  Contents
  
  2014-11-19 07:39:21,069  ít93282ù  DEBUG bulk.CdbRandomAccess -
  Objects
  left: 0 ,Status: -1=0 ,0=0 ,1=0 2=0 ,statusLevel=2
  2014-11-19 07:39:21,069  ít93282ù  INFO bulk.CdbBulkStatistics -
  Total
  time in core: 7066109 milliseconds
  
  
  However, when taking a thread dump of the storage service(or
  topology if domain mode) java process, you can see a deadlock in
  the javacore file;
  
  
  1LKDEADLOCK    Deadlock detected !!!
  NULL           ---------------------
  NULL
  2LKDEADLOCKTHR  Thread "RMI TCP Connection(8048)-10.119.10.166"
  (0x000000000148F700)
  3LKDEADLOCKWTR    is waiting for:
  4LKDEADLOCKMON      sys_mon_t:0x00002AAAD2635928 infl_mon_t:
  0x00002AAAD26359A0:
  4LKDEADLOCKOBJ
  com/ibm/cdb/topomgr/engine/TopologyStorageEngine@0x0000000699E3B
  6E8
  3LKDEADLOCKOWN    which is owned by:
  2LKDEADLOCKTHR  Thread "ChgWork-2" (0x00000000014C7A00)
  3LKDEADLOCKWTR    which is waiting for:
  4LKDEADLOCKMON      sys_mon_t:0x00002AAAD26379F8 infl_mon_t:
  0x00002AAAD2637A70:
  4LKDEADLOCKOBJ
  org/datanucleus/jdo/JDOPersistenceManager@0x00000006D054A528
  3LKDEADLOCKOWN    which is owned by:
  2LKDEADLOCKTHR  Thread "RMI TCP Connection(8048)-10.119.10.166"
  (0x000000000148F700)
  

Problem conclusion

• The fix for this APAR is contained in the following maintenance
  packages:
  | Fix Pack | 7.2.2-TIV-ITADDM-FP0004
  | Fix Pack | 7.3-TIV-ITADDM-FP0001
  Check the IBM Software Support web site for availability of the
  above maintenance packages.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  APP DEPENDENCY

• Fixed component ID

  5724N5500

Applicable component levels

• R722 PSY

     UP