IV40479: DEADLOCK OCCURS WHEN MULTIPLE THREADS TRY TO LOCK BROKER CONFIGURATION FILE LEADING TO HANGING.

IBM Tivoli Network Manager IP 3.9.0 Fix Pack 4, 3.9.0-TIV-ITNMIP-FP0004
IBM Tivoli Network Manager IP Edition 3.9.0 Fix Pack 5, 3.9.0-TIV-ITNMIP-FP0005

APAR status

• Closed as program error.

Error description

• This behavior occurs specially on Windows systems, where
  both ITNM and ITM agent for ITNM are installed, but might
  happen on other operating systems
  
  
  When deadlock occured , error messages like:
  
  Error: An unrecoverable error occurred. Please contact IBM
  Support.
  Fatal Error: Unable to lock mutex. found in file CRivMutex.cc at
  line 112 - Invalid argument in mutex created from
  CRivSockEngine.cc
  Application terminating due to error.
  
  
  are generated in logs associated with processes that are trying
  to lock the config file.
  

Local fix

• N/A
  

Problem summary

• ****************************************************************
  * USERS AFFECTED:                                              *
  * Seen on windows, may be multiplatform                        *
  ****************************************************************
  * PROBLEM DESCRIPTION:                                         *
  * Locking of ITNM due to deadlock when two threads in a        *
  * process try to lock the broker.cfg.lock file                 *
  *  The log with two separate threads showing "Broker           *
  * disconnection detected, reconnecting ..." was the one that   *
  * locked:                                                      *
  *                                                              *
  * 2013-07-24T09:26:53: Warning: W-MOM-001-007: [4151174864t]   *
  * Broker error on receive:                                     *
  * 2013-07-24T09:26:53: Warning: W-MOM-001-031: [32684912t]     *
  * Broker disconnection detected, reconnecting ...              *
  * 2013-07-24T09:26:53: Warning: W-MOM-001-031: [324234233t]    *
  * Broker disconnection detected, reconnecting ...              *
  ****************************************************************
  * RECOMMENDATION:                                              *
  * fix pack 3.9.0-ITNMIP-FP0004                                 *
  ****************************************************************
  Entire ITNM application locks up and needing a reboot.
  In the log files one of the logs, could be any one,  would show
  broker disconnection and reconnecting.  The log with two
  separate threads showing "Broker disconnection detected,
  reconnecting ..." was the one that locked:
  
  2013-07-24T09:26:53: Warning: W-MOM-001-007: [4151174864t]
  Broker error on receive:
  2013-07-24T09:26:53: Warning: W-MOM-001-031: [32684912t] Broker
  disconnection detected, reconnecting ...
  2013-07-24T09:26:53: Warning: W-MOM-001-031: [324234233t] Broker
  disconnection detected, reconnecting ...
  

Problem conclusion

• fix pack 3.9.0-ITNMIP-FP0004
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  TIV NETWK MGR I

• Fixed component ID

  5724S4500

Applicable component levels

• R390 PSY

     UP