IT05165: DATA MOVEMENT OPERATION WITH PARALLEL REORG MIGHT CRASH THE TIVOLI STORAGE MANAGER SERVER

6.3.6.000-TIV-TSMRPT-AGENT-Linux
6.3.6.000-TIV-TSMRPT-AGENT-AIX
6.3.6.000-TIV-TSMRPT-AIX
6.3.6.000-TIV-TSMRPT-AGENT-Windows
6.3.6.000-TIV-TSMRPT-Linuxx86
6.3.6.000-TIV-TSMRPT-Linuxx86_64
6.3.6.000-TIV-TSMRPT-WindowsI32
6.3.6.000-TIV-TSMRPT-WindowsX64
6.3.6.000-TIV-TSMAC-WindowsX64
6.3.6.000-TIV-TSMAC-WindowsI32
6.3.6.000-TIV-TSMAC-SolarisSPARC
6.3.6.000-TIV-TSMAC-Linuxx86
6.3.6.000-TIV-TSMAC-Linuxs390x
6.3.6.000-TIV-TSMAC-AIX
6.3.6.000-TIV-TSMALL-SolarisSPARC
6.3.6.000-TIV-TSMALL-WindowsX64
6.3.6.000-TIV-TSMSTA-WindowsI32
6.3.6.000-TIV-TSMALL-HP-UX
6.3.6.000-TIV-TSMALL-Linuxppc64
6.3.6.000-TIV-TSMALL-AIX
6.3.6.000-TIV-TSMALL-Linuxs390x
6.3.6.000-TIV-TSMALL-Linuxx86_64
IBM Tivoli Storage Manager (TSM) server V7.1.1.x interim fix downloads
IBM Tivoli Storage Manager V6.3 Fix Pack 6 (6.3.6.000) Server Downloads

APAR status

• Closed as program error.

Error description

• The Tivoli Storage Manager server might consume the whole system
  memory (including paging space) and gets finally
  killed by the operating system.
  The problem occurs when the REORG holds a lock on a table used
  by a data movement operation
  resulting in an ANR1880W message and the cancellation of the
  current transaction. Afterwards
  the actlog is flooded by any quickly repeated messages that
  continue until the server runs out of memory.
  Only data movement operations involving copy storage pools could
  hit the issue.
  
  Customer/L2 Diagnostics:
  The actlog may show the following message sequence:
  08/31/14   10:05:32      ANR0984I Process 90 for BACKUP STORAGE
  POOL started in
                            the FOREGROUND at 10:05:32. (SESSION:
  172555, PROCESS: 90)
  [...]
  08/31/14   10:07:15      ANR1880W Server transaction was
  canceled
   because of a conflicting lock on table BF_AGGREGATED_BITFILES.
  (SESSION: 172555, PROCESS: 90)
  
  Afterwards the actlog is flooded with million repeating entries
  like:
  08/31/14   10:07:15      ANR0130E tbcli.c(8120): Server LOG
  space exhausted.(SESSION: 172555, PROCESS: 90)
  08/31/14   10:07:15      ANR0162W Supplemental database
  diagnostic information:  -1:40003:-99999
   ([IBM][CLI Driver] CLI0108E   Communication link failure.
  SQLSTATE=40003).  (SESSION:     172555, PROCESS: 90)
  [...]
  until the server runs out of memory. The million repeating
  entries in the actlog are characteristic for this APAR.
  
  The db2diag.log does not report an out of log space condition as
  mentioned in the actlog message.
  
  The "show alloc" output from the server shows that in this case
  the DfBackupPoolThread consumes a lot of memory
  for writing messages to the actlog:
     output.c line  4999: 37242322 entries for   24430963232 bytes
  (AllocDataBlock) thread 268673(DfBackupPoolThread parent=268670)
  
  
  Tivoli Storage Manager Versions Affected:
  Tivoli Storage Manager server: 6.2, 6.3, 7.1 on all platforms
  
  Initial Impact:
  Medium
  
  Additional Keywords:
  TSM, REORG, IC81261, storagepool, loop, pgsp_kill, oom killer
  

Local fix

• Do not run REORG and any data movement operation in parallel
  

Problem summary

• ****************************************************************
  * USERS AFFECTED:                                              *
  * All Tivoli Storage Manager server users.                     *
  ****************************************************************
  * PROBLEM DESCRIPTION:                                         *
  * See error description.                                       *
  ****************************************************************
  * RECOMMENDATION:                                              *
  * Apply fixing level when available. This                      *
  * problem is currently projected to be fixed                   *
  * in levels 6.3.6, 7.1.1.200 and 7.1.3. Note that this is      *
  * subject to change at the discretion of IBM.                  *
  ****************************************************************
  

Problem conclusion

• This problem was fixed.
  Affected platforms:  AIX, HP-UX, Solaris, Linux, and Windows.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  TSM SERVER

• Fixed component ID

  5698ISMSV

Applicable component levels

• R62A PSY

     UP

• R62H PSY

     UP

• R62L PSY

     UP

• R62S PSY

     UP

• R62W PSY

     UP

• R63A PSY

     UP

• R63H PSY

     UP

• R63L PSY

     UP

• R63S PSY

     UP

• R63W PSY

     UP

• R71A PSY

     UP

• R71H PSY

     UP

• R71L PSY

     UP

• R71S PSY

     UP

• R71W PSY

     UP