IBM Support

IT05165: DATA MOVEMENT OPERATION WITH PARALLEL REORG MIGHT CRASH THE TIVOLI STORAGE MANAGER SERVER

Subscribe

You can track all active APARs for this component.

APAR status

  • Closed as program error.

Error description

  • The Tivoli Storage Manager server might consume the whole system
    memory (including paging space) and gets finally
    killed by the operating system.
    The problem occurs when the REORG holds a lock on a table used
    by a data movement operation
    resulting in an ANR1880W message and the cancellation of the
    current transaction. Afterwards
    the actlog is flooded by any quickly repeated messages that
    continue until the server runs out of memory.
    Only data movement operations involving copy storage pools could
    hit the issue.
    
    Customer/L2 Diagnostics:
    The actlog may show the following message sequence:
    08/31/14   10:05:32      ANR0984I Process 90 for BACKUP STORAGE
    POOL started in
                              the FOREGROUND at 10:05:32. (SESSION:
    172555, PROCESS: 90)
    [...]
    08/31/14   10:07:15      ANR1880W Server transaction was
    canceled
     because of a conflicting lock on table BF_AGGREGATED_BITFILES.
    (SESSION: 172555, PROCESS: 90)
    
    Afterwards the actlog is flooded with million repeating entries
    like:
    08/31/14   10:07:15      ANR0130E tbcli.c(8120): Server LOG
    space exhausted.(SESSION: 172555, PROCESS: 90)
    08/31/14   10:07:15      ANR0162W Supplemental database
    diagnostic information:  -1:40003:-99999
     ([IBM][CLI Driver] CLI0108E   Communication link failure.
    SQLSTATE=40003).  (SESSION:     172555, PROCESS: 90)
    [...]
    until the server runs out of memory. The million repeating
    entries in the actlog are characteristic for this APAR.
    
    The db2diag.log does not report an out of log space condition as
    mentioned in the actlog message.
    
    The "show alloc" output from the server shows that in this case
    the DfBackupPoolThread consumes a lot of memory
    for writing messages to the actlog:
       output.c line  4999: 37242322 entries for   24430963232 bytes
    (AllocDataBlock) thread 268673(DfBackupPoolThread parent=268670)
    
    
    Tivoli Storage Manager Versions Affected:
    Tivoli Storage Manager server: 6.2, 6.3, 7.1 on all platforms
    
    Initial Impact:
    Medium
    
    Additional Keywords:
    TSM, REORG, IC81261, storagepool, loop, pgsp_kill, oom killer
    

Local fix

  • Do not run REORG and any data movement operation in parallel
    

Problem summary

  • ****************************************************************
    * USERS AFFECTED:                                              *
    * All Tivoli Storage Manager server users.                     *
    ****************************************************************
    * PROBLEM DESCRIPTION:                                         *
    * See error description.                                       *
    ****************************************************************
    * RECOMMENDATION:                                              *
    * Apply fixing level when available. This                      *
    * problem is currently projected to be fixed                   *
    * in levels 6.3.6, 7.1.1.200 and 7.1.3. Note that this is      *
    * subject to change at the discretion of IBM.                  *
    ****************************************************************
    

Problem conclusion

  • This problem was fixed.
    Affected platforms:  AIX, HP-UX, Solaris, Linux, and Windows.
    

Temporary fix

Comments

APAR Information

  • APAR number

    IT05165

  • Reported component name

    TSM SERVER

  • Reported component ID

    5698ISMSV

  • Reported release

    62A

  • Status

    CLOSED PER

  • PE

    NoPE

  • HIPER

    NoHIPER

  • Special Attention

    NoSpecatt / Xsystem

  • Submitted date

    2014-10-28

  • Closed date

    2014-11-19

  • Last modified date

    2015-01-13

  • APAR is sysrouted FROM one or more of the following:

  • APAR is sysrouted TO one or more of the following:

Fix information

  • Fixed component name

    TSM SERVER

  • Fixed component ID

    5698ISMSV

Applicable component levels

  • R62A PSY

       UP

  • R62H PSY

       UP

  • R62L PSY

       UP

  • R62S PSY

       UP

  • R62W PSY

       UP

  • R63A PSY

       UP

  • R63H PSY

       UP

  • R63L PSY

       UP

  • R63S PSY

       UP

  • R63W PSY

       UP

  • R71A PSY

       UP

  • R71H PSY

       UP

  • R71L PSY

       UP

  • R71S PSY

       UP

  • R71W PSY

       UP



Document information

More support for: Tivoli Storage Manager

Software version: 62A

Reference #: IT05165

Modified date: 13 January 2015