IBM Support

IT18356: IBM SPECTRUM PROTECT DATABASE GROWS IN INDEX FOR TABLE SD_CHUNK_LOCATIONS AFTER A FAILING ONLINE INDEX REORGANIZATION

Subscribe

You can track all active APARs for this component.

APAR status

  • Closed as program error.

Error description

  • Index for table SD_CHUNK_LOCATIONS grows very fast after online
    index reorganization fails e.g. due to a database deadlock,
    which can lead to the DB containers not having enough free space
    for the next online index reorganization. This will then fail
    with "DIA8312C Disk was full."
    
    For example:
    ACTLOG shows:
    ANR0317I Reorganization of indices for table SD_CHUNK_LOCATIONS
    started.
    ANR0162W Supplemental database diagnostic information:
    -911:40001:0 (32).
    ANR0318I Reorganization of indices for table SD_CHUNK_LOCATIONS
    ended with sqlCode -911.
    ANR0159E tbrsql.c(1450): Database deadlock detected on 379:12.
    
    db2diag.log shows:
    FUNCTION: DB2 UDB, relation data serv, sqlrreorg_indexes,
    probe:1000
    MESSAGE : ADM9504W  Index reorganization on table "TSMDB1
    .SD_CHUNK_LOCATIONS"
    (ID "4") and table space "DEDUPTBLSPACE3" (ID "22")
    failed on this database partition with SQLCODE "-911" reason
    code "2".
    
    FUNCTION: DB2 UDB, catalog services,
    sqlrlCatalogScan::postUpdateCacheHandling, probe:10
    RETCODE : ZRC=0x80100002=-2146435070=SQLP_LDED "Dead lock
    detected"
    DIA8002C A deadlock has occurred, rolling back transaction.
    
    
    The next online index reorganization can fail due to lack of
    space in the database containers.
    
    db2diag.log shows:
    FUNCTION: DB2 UDB, relation data serv, sqlrreorg_indexes,
    probe:400
    MESSAGE : ADM9501W  Index reorganization has started for table
    "TSMDB1.SD_CHUNK_LOCATIONS" (ID "4") and table space
    "DEDUPTBLSPACE3" (ID "22").
    
    
    FUNCTION: DB2 UDB, relation data serv, sqlrreorg_index_obj,
    probe:550
    MESSAGE : ADM9503W  Reorganizing index IID "1" (OBJECTID "4") in
    table space "DEDUPIDXSPACE3" (ID "23") for table
    "TSMDB1.SD_CHUNK_LOCATIONS" (ID "4") in table space
    "DEDUPTBLSPACE3" (ID "22").
    
    FUNCTION: DB2 UDB, buffer pool services, sqlbGrowNewStripeSet,
    probe:20
    MESSAGE : ZRC=0x850F000C=-2062614516=SQLO_DISK "Disk full."
              DIA8312C Disk was full.
    DATA #1 : String, 108 bytes
    An error caused the allocation of a new stripe set of containers
    to an automatic storage tablespace to fail.
    
    FUNCTION: DB2 UDB, buffer pool services, sqlbAllocateExtent,
    probe:980
    MESSAGE : ADM6044E  The DMS table space "DEDUPIDXSPACE3" (ID
    "23") is full.  If this is an autoresize or automatic storage
    DMS tablespace, the maximum table space size may have been
    reached or the existing containers or storage paths cannot grow
    any more. Additional space can be added to the table space by
    either adding new containers or extending existing ones using
    the ALTER TABLESPACE SQL statement. If this is an autoresize or
    automatic storage DMS table space, additional space can be added
    by adding containers to an autoresize table space or by adding
    new storage paths to the storage group it is using.
    
    
    FUNCTION: DB2 UDB, buffer pool services, sqlbObtainDataExtent,
    probe:800
    MESSAGE : ZRC=0x85020021=-2063466463=SQLB_END_OF_CONTAINER
              "DMS Container space full"
    DATA #1 : Object descriptor, PD_TYPE_SQLB_OBJECT_DESC, 104 bytes
        Obj: {pool:23;obj:4;type:33} Parent={22;4}
    
    FUNCTION: DB2 UDB, relation data serv, sqlrreorg_indexes,
    probe:1000
    MESSAGE : ADM9504W  Index reorganization on table "TSMDB1
    .SD_CHUNK_LOCATIONS"
              (ID "4") and table space "DEDUPTBLSPACE3" (ID "22")
    failed on this database partition with SQLCODE "-289" reason
    code "DEDUPIDXSPACE3".
    
    
    IBM Spectrum Protect Server Versions Affected:
    IBM Spectrum Protect  Version 7.1.3 and above on all supported
    platforms
    
    Initial Impact:
    High
    
    Additional Keywords:
    TSM, inline deduplication, DB2, reorg, IBM Spectrum Protect
    

Local fix

  • Under normal circumstances SD tables should not require offline
    reorganization. However as a local fix for this issue the
    offline reorganization of the SD_CHUNK_LOCATION table will
    address the issue for the given online index reorg failure.
    
    
    Perform offline index reorg for table SD_CHUNK_LOCATIONS as
    described in Technote:
    
    "Resolving and preventing issues related to database growth and
    degraded performance in Tivoli Storage Manager V7.1.1.200 and
    later servers"
    http://www-01.ibm.com/support/docview.wss?uid=swg21683633
    

Problem summary

  • ****************************************************************
    * USERS AFFECTED:                                              *
    * All IBM Spectrum Protect server users.                       *
    ****************************************************************
    * PROBLEM DESCRIPTION:                                         *
    * See error description.                                       *
    ****************************************************************
    * RECOMMENDATION:                                              *
    * Apply fixing level when available. This problem is currently *
    * projected to be fixed in levels 7.1.8 and 8.1.1. Note that   *
    * this is subject to change at the discretion of IBM.          *
    ****************************************************************
    

Problem conclusion

  • This problem was fixed.
    Affected platforms for reported release:  AIX, HP-UX, Solaris,
    Linux, and Windows.
    Platforms fixed:  AIX, Solaris, Linux, and Windows.
    

Temporary fix

Comments

APAR Information

  • APAR number

    IT18356

  • Reported component name

    TSM SERVER

  • Reported component ID

    5698ISMSV

  • Reported release

    71W

  • Status

    CLOSED PER

  • PE

    NoPE

  • HIPER

    NoHIPER

  • Special Attention

    NoSpecatt / Xsystem

  • Submitted date

    2016-12-15

  • Closed date

    2017-02-10

  • Last modified date

    2017-04-24

  • APAR is sysrouted FROM one or more of the following:

  • APAR is sysrouted TO one or more of the following:

Fix information

  • Fixed component name

    TSM SERVER

  • Fixed component ID

    5698ISMSV

Applicable component levels

  • R71A PSY

       UP

  • R71H PSY

       UP

  • R71L PSY

       UP

  • R71S PSY

       UP

  • R71W PSY

       UP

  • R81A PSY

       UP

  • R81L PSY

       UP

  • R81W PSY

       UP



Document information

More support for: Tivoli Storage Manager

Software version: 7.1.3

Reference #: IT18356

Modified date: 24 April 2017