IC91190: SERVER TERMINATES DUE TO ACTIVE LOG FILLING WHEN LONG RUNNING REORG IS PROCESSING. SQLCODE "-964" IN DB2DIAG.LOG.

IBM Tivoli Storage Manager V6.3 Fix Pack 5 (6.3.5.000) Server Downloads

APAR status

• Closed as program error.

Error description

• The Tivoli Storage Manager server will initiate reorg processing
  against DB2 database indices. In rare cases where the index
  being processing is very large, the reorg activity can consume a
  great amount of space while also pinning the log. When there is
  significant other workload on the server the active log usage
  can grow to the point of filling the log. If/when the active log
  becomes full the Tivoli Storage Manager server will terminate.
  The excessive use of active log space by reorg processing is a
  DB2 issue being addressed by DB2 APAR IC91020. When the DB2 fix
  for APAR IC91020 is available, this Tivoli Storage Manager APAR
  will provide the DB2 fix within the Tivoli Storage Manager
  embedded DB2 code.
  To determine if the server has terminated due to this issue the
  following message will be seen in the Tivoli Storage Manager
  activity log at some point prior to the failure (may be hours
  before):
  ANR0317I Reorganization for indices for table <table name>
  started.
  The db2diag.log will have the following entry for this same
  table at some point prior to the server terminating:
  2013-03-06-19.53.30.825000-360 E8336134F643       LEVEL: Warning
  PID     : 10256                TID  : 8256        PROC :
  db2syscs.exe
  INSTANCE: SERVER1              NODE : 000         DB   : TSMDB1
  APPHDL  : 0-55195              APPID:
  *LOCAL.SERVER1.130306183039
  AUTHID  : TSMINST1
  EDUID   : 8256                 EDUNAME: db2agent (TSMDB1) 0
  FUNCTION: DB2 UDB, relation data serv, sqlrreorg_indexes,
  probe:1000
  MESSAGE : ADM9504W  Index reorganization on table "<table name>"
            (ID "24") and table space "USERSPACE1" (ID "2") failed
            on this database partition with SQLCODE "-964"
            reason code "".
  Note that the -964 reason code is the key that DB2 APAR IC91020
  is the culprit.
  There is NO definitive message in the server activity log
  indicating the server is terminating for a log full condition.
  Tivoli Storage Manager Versions Affected:  6.2, 6.3 (DB2 9.7)
  Also note that server APAR IC90985 may compound this issue as it
  address repetative reorg processing against indices for
  BF_BITFILE_EXTENTS
  Initial Impact: Medium
  
  Additional Keywords: TSM reorganize failure crash stopped ended
  

Local fix

Problem summary

• ****************************************************************
  * USERS AFFECTED: All Tivoli Storage Manager server users.     *
  *                                                              *
  ****************************************************************
  * PROBLEM DESCRIPTION:  See error description.                 *
  *                                                              *
  ****************************************************************
  * RECOMMENDATION: Apply fixing level when available. This      *
  *                 problem                                      *
  *                 is currently projected to be fixed           *
  *                 in levels 6.2.6, 6.3.5, and 7.1.1.           *
  *                 Note that this                               *
  *                 is subject to change at the                  *
  *                 discretion of IBM.                           *
  *                                                              *
  ****************************************************************
  
  You might experience the issue that is documented
  in this APAR if both of the following conditions are true:
  1) After you run the DB2 REORGCHK command, the output for
  Formula 7 or Formula 8 is true for the BF_BITFILE_EXTENTS
  table. Typically, this occurs only when a large amount of
  deduplicated data is deleted from the server. For example,
  deleting one or more nodes with large amounts of deduplicated
  data (in comparison with the total amount of deduplicated
  data on the server) could cause this to occur.
  2) The BF_BITFILE_EXTENTS table is large, and other concurrent
  work on the server is causing the active log to become full
  while on-demand index reorganization is running.
  
  Because of the variability of deduplicated data and the active
  log, the conditions cannot be quantified in more detail.
  

Problem conclusion

• This problem was fixed.                                        A
  ffected platforms:  AIX, HP-UX, Solaris, Linux and Windows.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  TSM SERVER

• Fixed component ID

  5698ISMSV

Applicable component levels

• R62A PSY

     UP

• R62H PSY

     UP

• R62L PSY

     UP

• R62S PSY

     UP

• R62W PSY

     UP

• R63A PSY

     UP

• R63H PSY

     UP

• R63L PSY

     UP

• R63S PSY

     UP

• R63W PSY

     UP