IBM Support

IT04997: VERY LARGE DISK STORAGE POOLS MAY RESULT IN STARTUP DELAYS OBSERVED FOR THE TIVOLI STORAGE MANAGER SERVER

Subscribe

You can track all active APARs for this component.

APAR status

  • Closed as program error.

Error description

  • If a Tivoli Storage Manager server is configured with very large
    random access (disk) storage pools, a server start might show a
    delay.
    
    Under the APAR condition, the call stack for the admStartServer
    thread seems to be hung in bfInit(). pstack coomand output
    collected several times will look similar to:
    
     ffffffff7c9dcf48 semsys   (2, 29000079, ffffffff7fffbde4, 1,
    2e8c8)
     ffffffff7aec7b9c
    __1cMsqlccipcrecv6FpnRSQdDLCC_COMHANDLE_T_pnMSQdDLCC_COND_T__h_
    (ffffffff76026ce0, ffffffff76026ba0, 19580000, 0, 0,
    ffffffff76027164) + 6fc
     ffffffff7aecd6fc sqlccrecv (ffffffff76026ce0, ffffffff76027100,
    10004, ffffffff7fffc1ec, 6, 90) + 314
     ffffffff7b12e370 __1cMsqljcReceive6FpnKsqljCmnMgr__i_
    (ffffffff76026978, 0, 0, f000, ffffffff7fffc1e0, 19b00000) + 130
     ffffffff7b198ec4
    __1cPsqljrDrdaArOpen6FpnOdb2UCinterface_pnPdb2UCCursorInfo__i_
    (ffffffff74600090, ffffffff19800620, 0, 0, 0, ffffffff7c451778)
    + 28c
     ffffffff7ab134ec
    __1cHcsmOpen6FpnOdb2UCinterface_pnPdb2UCCursorInfo__i_
    (ffffffff74600090, ffffffff7b198c38, 0, 10070dc, 0, 19f00000) +
    9c
     ffffffff7a7ee074
    __1cLCLI_sqlOpen6FpnRCLI_STATEMENTINFO_pnTCLI_ERRORHEADERINFO__h
    _ (ffffffff19800080, ffffffff19800090, 1, ffffffff19800620, 3,
    0) + ad4
     ffffffff7a8eaa9c
    __1cLSQdDLExecute26FpnRCLI_STATEMENTINFO_pnTCLI_ERRORHEADERINFO_
    _h_ (ffffffff19800080, ffffffff19800090, 2ce0, 1800, 1000, 2c00)
    + 18d4
     ffffffff7a8db834 SQLExecute (1000000, 2c00, 19530000, 0, 0,
    2cd4) + 11d4
     0000000101833c14 tbRegExecEx (104ceec48, 0, ffffffff7fffcdf0,
    ffffffff7fffcddc, ffffffff7fffcde8, ffffffff7fffcdf8) + 17e4
     0000000101da618c dsObtainFreeListForVol (104904968, 17096,
    c34fff, 104ac1ac4, 104ac1ad0, 103a40988) + 804
     000000010185f988 ObtainFreeList (104904968, 104ac1a88, 0, 3,
    7fffffff, e) + 418
     000000010185bd74 bvOpen (104904968, 17096, ffffffff7fffd290, 1,
    10370df30, e) + 5d4
     0000000101dc6a20 DsGetPoolStatus (4, ffffffff7fffd798,
    ffffffff7fffd7a0, ffffffff7fffd7b8, ffffffff7fffd7b0, 0) + bd0
     0000000101e29998 ssGetNextPool (ffffffff7fffd5e0,
    ffffffff7fffd5e0, 1, ffffffff7fffd5e0, 0, 1047373a8) + 12b0
     000000010090d258 BfInitDedup (104bebd98, 306, 104bebd98,
    104be3258, 0, 104be3258) + 488
     000000010070d590 bfInit (1, 0, 173a, 4753495a4500, ff0000,
    8080808080808080) + 1020
     00000001001f558c StartServer (0, 0, ffffffff7fffeeac,
    ffffffff7fffeea8, 102e1c718, 1046fc8a8) + 2354
     00000001001e564c admStartServer (0, 0, 527, 102623cb0,
    104698699, 104698298) + 1dc
     000000010007c8d0 main (1, ffffffff7ffff888, ffffffff7ffff898,
    ffffffff7c94c920, ffffffff79300b40, ffffffff79700200) + 27a0
     00000001000779dc _start (0, 0, 0, 0, 0, 0) + 17c
    
    A trace with traceflags PVR MMS will show the delay as with the
    following example:
    
    15:51:18.781 [26][mmsshr.c][845][MmsLibraryPollingThread]:Timed
    out
    waiting for session manager initialization.
    16:16:18.841 [26][mmsshr.c][845][MmsLibraryPollingThread]:Timed
    out
    waiting for session manager initialization.
    16:41:18.902 [26][mmsshr.c][845][MmsLibraryPollingThread]:Timed
    out
    waiting for session manager initialization.
    16:51:17.170 [1][pvrclass.c][5644]
    [pvrIsFileDevClass]:pvrIsFileDevClass: device class
    <SOMEDEVICECLASSis not File
    
    The trace above documnets the initialization of a disk storage
    pool exceeding 20TB in size.
    
      Tivoli Storage Manager Versions Affected: all supported
    versions
      Initial Impact: Medium
      Additional Keywords: bitvector bfo hang stgpool zz62 zz63 zz64
    zz71
    

Local fix

  • reduce the size of the disk stoarge pool or use file device
    storage pool
    When you look for a recommendation on how much to lower the disk
    storage pool to, as a rule of thumb a disk storage pool should
    only be large enough to contain one nights worth of backups and
    then migrate off to sequential media.
    In addition, larger backups (for example, TDP type backups or
    system image type backups) should go direct to sequential media
    skipping the disk storage pool.
    

Problem summary

  • ****************************************************************
    * USERS AFFECTED:                                              *
    * All Tivoli Storage Manager server users.                     *
    ****************************************************************
    * PROBLEM DESCRIPTION:                                         *
    * See error description.                                       *
    ****************************************************************
    * RECOMMENDATION:                                              *
    * Apply fixing level when available. This problem is currently *
    * projected to be fixed in level 7.1.5. Note that this is      *
    * subject to change at the discretion of IBM.                  *
    *                                                              *
    * New FFDC entries are being created to help determine how     *
    * long it takes to build the bitvector for each of the DISK    *
    * storage pool volumes.  This processing only happens once on  *
    * server startup or when a new volume is created.              *
    *                                                              *
    * The following are the sample entries created for each        *
    * volume:                                                      *
    *                                                              *
    * [01-29-2016 14:47:11.297][ FFDC_GENERAL_SERVER_ERROR ]:      *
    * (dsalloc.c:574) Building freelist for volid 110              *
    * [01-29-2016 14:47:11.430][ FFDC_GENERAL_SERVER_ERROR ]:      *
    * (dsalloc.c:869) Built freelist for volid 110 from 10         *
    * database rows.                                               *
    *                                                              *
    * A large number of entries on a volume could indicate         *
    * fragmented disk storage, client underallocation into the     *
    * storage pool or client data that is not aggregated such as   *
    * API applications that use single file transactions.  For     *
    * this type of API data a file pool or a container pool should *
    * be used instead of a disk storage pool.                      *
    *                                                              *
    * The following can be used to get the volume name that        *
    * matches the volume id from a db2 command line prompt:        *
    *                                                              *
    * db2 connect to tsmdb1                                        *
    * C:\Program Files\Tivoli\TSM\server>db2 -x "select volname    *
    * from tsmdb1.ss_volume_names where volid=110"                 *
    *                                                              *
    * Sample output:                                               *
    * C:\PROGRAM FILES\TIVOLI\TSM\SERVER\DISKVOL000                *
    ****************************************************************
    

Problem conclusion

  • This problem was fixed.
    Affected platforms:  AIX, HP-UX, Solaris, Linux, and Windows.
    

Temporary fix

Comments

APAR Information

  • APAR number

    IT04997

  • Reported component name

    TSM SERVER

  • Reported component ID

    5698ISMSV

  • Reported release

    63W

  • Status

    CLOSED PER

  • PE

    NoPE

  • HIPER

    NoHIPER

  • Special Attention

    NoSpecatt / Xsystem

  • Submitted date

    2014-10-17

  • Closed date

    2016-02-01

  • Last modified date

    2016-02-01

  • APAR is sysrouted FROM one or more of the following:

  • APAR is sysrouted TO one or more of the following:

Fix information

  • Fixed component name

    TSM SERVER

  • Fixed component ID

    5698ISMSV

Applicable component levels

  • R71A PSY

       UP

  • R71H PSY

       UP

  • R71L PSY

       UP

  • R71S PSY

       UP

  • R71W PSY

       UP



Document information

More support for: Tivoli Storage Manager

Software version: 63W

Reference #: IT04997

Modified date: 01 February 2016