IV52877: DEADLOCK DETECTION OCCURS WHILE CACHING NETWORK VIEWS(MYSQL ONLY)

IBM Tivoli Network Manager IP 3.9.0 Fix Pack 4, 3.9.0-TIV-ITNMIP-FP0004
IBM Tivoli Network Manager IP Edition 3.9.0 Fix Pack 5, 3.9.0-TIV-ITNMIP-FP0005

APAR status

• Closed as program error.

Error description

• v3.9-FP3 topped with Interim fix - Mysql being NCIM.
  
  Deadlock detections noticed in following scenario:
  While loading the networkView Tree for the very first time, the
  Views in the tree are being registered for events. Registering
  views for the events causes to cache the Views. During caching
  process, following deadlock detections are noticed:
  
  Excerpt from ncp_topoviz.0.trace:
  <snip>
  [2013-11-18T09:57:12.057] [pool-80-thread-3]
  NcimDatabaseDAOImpl. aggregateTempEntities(Connection conn,
  Set<Integer> entityIds) FINER: Batched 0 insertions for entity
  aggregation in 0 ms.
  [2013-11-18T09:57:12.056] [pool-80-thread-1]
  NetworkViewDAOImpl.storeCache(Connection, NetworkViewBean)
  SEVERE: Failed to store the new cache view for viewId 1856
  java.sql.SQLException: Deadlock found when trying to get lock;
  try restarting transaction
          at
  com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2921)
          at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1570)
          at
  com.mysql.jdbc.ServerPreparedStatement.serverExecute(ServerPrepa
  redStatement.java:1085)
          at
  com.mysql.jdbc.ServerPreparedStatement.executeInternal(ServerPre
  paredStatement.java:670)
          at
  com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement
  .java:1159)
          at
  com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement
  .java:1076)
          at
  com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement
  .java:1061)
          at
  com.micromuse.topoviz.server.dblayer.NetworkViewDAOImpl.storeCac
  heInserts(NetworkViewDAOImpl.java:3978)
          at
  com.micromuse.topoviz.server.dblayer.NetworkViewDAOImpl.storeCac
  he(NetworkViewDAOImpl.java:3816)
          at
  com.micromuse.topoviz.server.dblayer.NetworkViewDAOImpl.getCache
  View(NetworkViewDAOImpl.java:4214)
          at
  com.micromuse.topoviz.server.dblayer.NetworkViewDAOImpl.cacheVie
  w(NetworkViewDAOImpl.java:4340)
          at
  com.micromuse.topoviz.server.dblayer.NetworkViewDAOImpl.cacheVie
  w(NetworkViewDAOImpl.java:4326)
          at
  com.micromuse.topoviz.server.dblayer.NetworkViewDAOImpl.getEntit
  yIdsForView(NetworkViewDAOImpl.java:2654)
          at
  com.micromuse.topoviz.server.dblayer.NetworkViewDAOImpl.getEntit
  yIdsForViewRecursive(NetworkViewDAOImpl.java:2699)
          at
  com.micromuse.topoviz.webtop.worker.ViewRegistrationExecutor.exe
  cute(ViewRegistrationExecutor.java:89)
          at
  com.micromuse.topoviz.webtop.worker.BaseRegistrationExecutor.run
  (BaseRegistrationExecutor.java:108)
          at
  java.util.concurrent.Executors$RunnableAdapter.call(Executors.ja
  va:452)
          at
  java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:31
  4)
          at
  java.util.concurrent.FutureTask.run(FutureTask.java:149)
          at
  java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoo
  lExecutor.java:897)
          at
  java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExe
  cutor.java:919)
          at java.lang.Thread.run(Thread.java:736)
  
  [2013-11-18T09:57:12.057] [pool-80-thread-3]
  NcimDatabaseDAOImpl. aggregateTempEntities(Connection conn,
  Set<Integer> entityIds) FINER: Leave: 0 ms.
  </snip>
  

Local fix

• 1) Stop TIP
  2) Stop all ITNM domains (ncp processes)
  3) Stop NCIM database(Mysql)
  4) Edit my.cnf (if no file is found, create one in /etc folder)
  and add following line: transaction-isolation=READ-COMMITTED
  
  The file context can be just below:
  # cat /etc/my.cnf
  [mysqld]
  transaction-isolation=READ-COMMITTED
  
  5) Restart mysql and verify the tx-isolation setting i.e.
  
  mysql> SELECT @@GLOBAL.tx_isolation, @@tx_isolation;
  +-----------------------+----------------+
  | @@GLOBAL.tx_isolation | @@tx_isolation |
  +-----------------------+----------------+
  | READ-COMMITTED        | READ-COMMITTED |
  +-----------------------+----------------+
  1 row in set (0.00 sec)
  
  If your results match above, then continue below steps:
  
  6) Delete networkView cache from ncpgui database (i.e. - delete
  from ncpgui.networkViewSumr;)
  7) Restart ITNM domain/s
  8) Restart TIP
  

Problem summary

• ****************************************************************
  * USERS AFFECTED:                                              *
  * ITNM 3.9 users                                               *
  ****************************************************************
  * PROBLEM DESCRIPTION:                                         *
  * Deadlock detections noticed in following scenario:           *
  * While loading the networkView Tree for the very first time,  *
  * the                                                          *
  * Views in the tree are being registered for events.           *
  * Registering                                                  *
  * views for the events causes to cache the Views.              *
  ****************************************************************
  * RECOMMENDATION:                                              *
  * Upgrade to ITNM-IP | fix pack | 3.9.0-ITNMIP-FP0004          *
  ****************************************************************
  

Problem conclusion

• Errors and exception are coming from the fact that the MYSQL DB
  is getting deadlock problems.
  Fix will be available in FP0004
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  TIV NETWK MGR I

• Fixed component ID

  5724S4500

Applicable component levels

• R390 PSN

     UP

• R390 PSY

     UP

• R401 PSN

     UP

• R401 PSY

     UP