IT19596: "REPAIR STGPOOL" FROM LOCAL CONTAINER-COPY POOL IS VERY SLOW WHEN MULTIPLE PRIMARY DIRECTORY-CONTAINER STORAGE POOLS EXISTS

7.1.9.000-TIV-TSMCMS-Windows64
7.1.9.000-TIV-TSMCMS-Windows32
7.1.9.000-TIV-TSMCMS-Linuxx64
7.1.9.000-TIV-TSMOC-LinuxS390
7.1.9.000-TIV-TSMOC-LinuxPPC64
7.1.9.000-TIV-TSMOC-Windows
7.1.9.000-TIV-TSMOC-Linuxx64
7.1.9.000-TIV-TSMOC-AIX
7.1.9.000-TIV-TSMSRV-WIN
7.1.9.000-TIV-TSMSRV-SolarisSPARC
7.1.9.000-TIV-TSMSRV-Linuxx86_64
7.1.9.000-TIV-TSMSRV-Linuxs390x
7.1.9.000-TIV-TSMSRV-Linuxppc64
7.1.9.000-TIV-TSMSRV-HP-UX
7.1.9.000-TIV-TSMSRV-AIX
8.1.5.000-IBM-SPCMS-WindowsX64
8.1.5.000-IBM-SPCMS-WindowsI32
8.1.5.000-IBM-SPCMS-Linuxx86_64
8.1.5.000-IBM-SPOC-WindowsX64
8.1.5.000-IBM-SPOC-Linuxx86_64
8.1.5.000-IBM-SPOC-Linuxs390x
8.1.5.000-IBM-SPOC-LinuxPPC64le
8.1.5.000-IBM-SPOC-AIX
8.1.5.000-IBM-SPSRV-WindowsX64
8.1.5.000-IBM-SPSRV-Linuxx86_64
8.1.5.000-IBM-SPSRV-Linuxs390x
8.1.5.000-IBM-SPSRV-Linuxppc64le
8.1.5.000-IBM-SPSRV-AIX
IBM Spectrum Protect Server V8.1 Fix Pack 5 (V8.1.5) Downloads
IBM Spectrum Protect Server V7.1 Fix Pack 9 (7.1.9.000) Downloads

APAR status

• Closed as program error.

Error description

• When a directory-container storage pool needs to be repaired,
  for example if the full storage pool or a complete storage pool
   directory is being lost, a "repair stgpool" as started from
  the (local) container-copy storage pool is dramatically slow
  (for instance it has been seen that only 3 GB have been
  restored in 3 days).
  That  only happen if there are several primary
  directory-containers storage pools defined to the server, and if
  the number of extents to repair is very large.
  
  
  IBM Spectrum Protect Versions Affected: 7.1.7.0 and 8.1.0.0  on
    all supported platforms
  
  
  Customer/L2 Diagnostics :
  
  
  - A "show threads" output will show the  following stack almost
    all the time for each worker thread:
  
  Thread 235, ID 8972 (0x230c): SdFormSegmentsWithinBatch,
   procToken=524, sessToken=2232
   Parent=225, result=0, joining=0, detached=0, zombie=0,
   session=0
     Holding mutex 77654368 (&txnP->mutex)
  
     Stack trace:
     000000000ED206FA Unknown
     00007FF90BF01118 WaitForSingleObjectEx()+98
     00007FF90602E1FD sqloWaitInterrupt()+2ed
     00007FF905F89B12 sqloSSemP()+42
     00007FF904A03947 sqlccIPCWaitForReceive()+3397
     00007FF904A08605 sqlccrecv()+195
     00007FF904BE297C sqljcReadGetPtrInt()+26dc
     00007FF904C13E48 sqljrDrdaArExecute()+518
     00007FF904F92892 CLI_CheckLicense()+95d2
     00007FF904E18224 SQLExecute()+3954
     00007FF904E14D6C SQLExecute()+49c
     00007FF8E2BB8616 RdbPrepareAndExecuteStmt()+746
     00007FF8E2BB7909 RdbCliUpdate()+2a9
     00007FF8E2BAC752 tbCliSRUpd()+362
     00007FF8E2C6B5B5 sdRepairReconOrder()+425
     00007FF8E2C767F8 LocalCatalogRepairBatch()+1f8
     00007FF8E2C75C2C LocalFlushRepairTxn()+10c
     00007FF8E2C763F5 LocalAcquireSegmentChunkDesc()+225
     00007FF8E2C6EE4C SdFormSegmentsWithinBatch()+33c
     00007FF8E2FC1121 PcConsumerThread()+141
     00007FF8E2517881 startThread()+141
     00007FF8EDA14F7F beginthreadex()+107
     00007FF8EDA15126 endthreadex()+192
     00007FF90E0A13D2 BaseThreadInitThunk()+22
     00007FF90ECA54E4 RtlUserThreadStart()+34
  
      Thread context:
  
     COMMAND: REPAIR STGPOOL
     COMMMETHOD: Tcp/Ip
     PROCESS_NUMBER: 001
     PROCESS_DESC: Repair Stgpool
     THREAD_TYPE: PROCESS
     SESSION_TYPE: ADMIN
     ADMIN_NAME: XXX
  
  
  - An instrumentation trace will show:
  
  Thread 235 SdFormSegmentsWithinBatch (Win Thread ID 8972)
  16:06:24.616-->16:26:58.516
  Operation     Count Tottime  Avgtime Mintime Maxtime InstTput
                                                          Total KB
  ----------------------------------------------------------------
  DB2 Updat Exec  76  1233.885 16.235    2.734  18.157
  DB2 Reg Exec    75     0.015  0.000    0.000   0.015
  Unknown                0.000
  ----------------------------------------------------------------
  Total               1233.900
  
  
  
  Initial Impact: Medium
  
  Additional Keywords:   TSM container restore stg copy
  

Local fix

• - Contact IBM support if a workaround is needed before the
    official correction.
  

Problem summary

• ****************************************************************
  * USERS AFFECTED:                                              *
  * All IBM Spectrum Protect server users.                       *
  ****************************************************************
  * PROBLEM DESCRIPTION:                                         *
  * See error description.                                       *
  ****************************************************************
  * RECOMMENDATION:                                              *
  * Apply fixing level when available. This problem is currently *
  * projected to be fixed in levels 7.1.9 and 8.1.5. Note that   *
  * this is subject to change at the discretion of IBM.          *
  ****************************************************************
  

Problem conclusion

• This problem was fixed.
  Affected platforms for reported release:  AIX, HP-UX, Solaris,
  Linux, and Windows.
  Platforms fixed:  AIX, Linux, and Windows.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  TSM SERVER

• Fixed component ID

  5698ISMSV

Applicable component levels