IC98834: SRA THREADS BECOME DEADLOCKED

Refresh Pack 5.2.2 (June 2014) for Tivoli Storage Productivity Center
Fix Pack 5.1.1.5 (July 2014) for Tivoli Storage Productivity Center
Refresh Pack 5.2.3 (August 2014) for Tivoli Storage Productivity Center
Fix Pack 5.2.4 (November 2014) for Tivoli Storage Productivity Center
Fix Pack 5.2.4.1 (December 2014) for Tivoli Storage Productivity Center
Refresh Pack 5.2.5 (March 2015) for Tivoli Storage Productivity Center (withdrawn)
Fix Pack 5.1.1.6 (March 2015) for Tivoli Storage Productivity Center
Fix Pack 5.2.5.1 (April 2015) for Tivoli Storage Productivity Center (withdrawn)
Refresh Pack 5.2.6 (June 2015) for Tivoli Storage Productivity Center
Refresh Pack 5.2.7 (August 2015) for Tivoli Storage Productivity Center
Fix Pack 5.1.1.9 (October 2015) for Tivoli Storage Productivity Center
IBM Spectrum Control V5.2.8 (December 2015)
IBM Spectrum Control V5.2.9 (February 2016)
IBM Spectrum Control V5.2.10 (May 2016)
IBM Spectrum Control V5.2.10.1 (July 2016)
IBM Spectrum Control V5.2.11 (August 2016)
Fix Pack 5.1.1.12 (October 2016) for Tivoli Storage Productivity Center
Fix Pack 5.1.1.13 (February 2017) for Tivoli Storage Productivity Center
Fix Pack 5.1.1.14 (June 2017) for Tivoli Storage Productivity Center
Fix Pack 5.1.1.15 (Sept 2017) for Tivoli Storage Productivity Center
IBM Spectrum Control V5.2.12 (November 2016)
IBM Spectrum Control V5.2.13 (March 2017)
IBM Spectrum Control V5.2.14 (May 2017)
IBM Spectrum Control V5.2.15 (August 2017)
IBM Spectrum Control V5.2.15.2 (November 2017)
IBM Spectrum Control V5.2.16 (March 2018)
IBM Spectrum Control V5.2.17 (May 2018)
Fix Pack 5.1.1.8 (July 2015) for Tivoli Storage Productivity Center
Fix Pack 5.2.7.1 (February 2016) for Tivoli Storage Productivity Center

APAR status

• Closed as program error.

Error description

• The root cause exists for all SRAs as described below. However,
  the symptom was seen to surface in the following scenario:
  After physically removing the SRAs from the environment, it is
  not possible
  to remove them from TPC. In fact, removing from the TPC GUI
  appears to be successful
  and no errors are issued, but the SRAs simply do not go away.
  Further, it is observed that
  the removed servers slowly disappear from the TPC GUI, but this
  may take several days or more
  (even months) to complete.
  The root cause is a deadlock in the database. In particular, one
  thread is launched for each agent
  selected for deletion. In some cases those threads are
  deadlocking each other:
  """"""""""""""""""""""""""""""" db2diag.log:
    com.ibm.db2.jcc.am.SqlTransactionRollbackException: DB2 SQL
  Error: SQLCODE=-911, SQLSTATE=40001,
  SQLERRMC=2, DRIVER=3.64.114
    [...]
          at
  com.tivoli.itsrm.server.UninstallAgentDriver.run(UninstallAgentD
  river.java:160)
  com.tivoli.itsrm.TStorm.repository.RepositoryAccess delete
  Thread-982
  """""""""""""""""""""""""""""""
  This exception occurs for all the agents that do not disappear.
  One error visible in ServerTrace.trace is the following:
  """"""""""""""""""""""""""
  014-01-15 10:54:43.080-0500 EXCEPTION
    com.tivoli.itsrm.common.GeneralSQLException: SRV0081E
    [...]
    at
  com.tivoli.itsrm.server.UninstallAgentDriver.remove(UninstallAge
  ntDriver.java:393)
    [...]
  com.tivoli.itsrm.TStorm.server.svp.ProbeResultsHndlr
  deleteComputer Thread-990
  """"""""""""""""""""""""""
  

Local fix

• Iteratively manage SRAs in groups of four or less.Contact suppo
  rt for patch or upgrade to newest release.
  

Problem summary

• ****************************************************************
  * USERS AFFECTED:                                              *
  * TPC 5.1.1.x or 5.2.0 users trying to remove many SRAs at     *
  * once.                                                        *
  ****************************************************************
  * PROBLEM DESCRIPTION:                                         *
  * See Error Description.                                       *
  ****************************************************************
  * RECOMMENDATION:                                              *
  * Apply fix maintenance when available.                        *
  ****************************************************************
  

Problem conclusion

• The agents will be deleted serially to avoid database tables
  deadlocks.  The fix for this APAR is targeted for the following
  maintenance package:
  
  | fix pack | 5.1.1-TIV-TPC-FP0005 - target July 2014
  | refresh pack | 5.2-TIV-TPC-RP0001 - target March 2014
  
  http://www-01.ibm.com/support/docview.wss?&uid=swg21320822
  
  The target dates for future refresh packs or fix packs do not
  represent a formal commitment by IBM. The dates are subject to
  change without notice.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Modules/Macros

• Data
  

Fix information

• Fixed component name

  TPC

• Fixed component ID

  5608TPC00

Applicable component levels

• R510 PSY

     UP

• R511 PSY

     UP

• R520 PSY

     UP