IT25776: SLOW PERFORMANCE DURING LOCAL PROTECT STGPOOL WHEN USING A CONTAINER-COPY POOL WITH PROTECTPROCESS=1

8.1.7.000-IBM-SPSRV-Linuxppc64le
8.1.7.000-IBM-SPCMS-WindowsX64
8.1.7.000-IBM-SPCMS-WindowsI32
8.1.7.000-IBM-SPCMS-Linuxx86_64
8.1.7.000-IBM-SPOC-WindowsX64
8.1.7.000-IBM-SPOC-Linuxx86_64
8.1.7.000-IBM-SPOC-Linuxs390x
8.1.7.000-IBM-SPOC-LinuxPPC64le
8.1.7.000-IBM-SPOC-AIX
8.1.7.000-IBM-SPSRV-WindowsX64
8.1.7.000-IBM-SPSRV-Linuxx86_64
8.1.7.000-IBM-SPSRV-Linuxs390x
8.1.7.000-IBM-SPSRV-AIX
8.1.1.300-IBM-SPSRV-WindowsX64
8.1.1.300-IBM-SPSRV-Linuxx86_64
8.1.1.300-IBM-SPSRV-Linuxs390x
8.1.1.300-IBM-SPSRV-Linuxppc64le
8.1.1.300-IBM-SPSRV-AIX
IBM Spectrum Protect Server V8.1.1.X interim fix downloads
IBM Spectrum Protect Server V8.1 Fix Pack 7 (V8.1.7) Downloads

APAR status

• Closed as program error.

Error description

• Error Description:  Local protect stgpool to a container-copy
  pool with PROTECTPRocess=1 will write to only one tape drive at
  a time.  However, it will also limit most processing to a single
   thread.  The local protect stgpool's initial analysis generally
   needs more threads than 1 to keep up with the I/O capabilities
  of most tape drives.  One way to increase the amount of
  parallelism for this initial analysis is to use the server
  option LOCALPROTCNTRCONS (Local Protect Container Consumer).
  The default value is 0, which indicates to use the same number
  as specified by the PROTECTPRocess value specified in the
  storage pool definition.  If the PROTECTPRocess value is 1-3, it
   is worthwhile to have more parallelism during the initial
  analysis.  Thus, the number of threads on the initial analysis
  should not necessarily be the same number as the PROTECTPRocess.
    This can affect all phases of local protect including
  deletion, ingest, and reclamation.
  
  Spectrum Protect Versions Affected: 7.1.x and 8.1.x on all
  supported platforms
  
  Customer/L2 Diagnostics: You can use the QUERY PROCESS output to
  monitor the "Extents deleted: xxxxx of xxxxx" field for the
  local protect process. That will indicate that local protect is
  in the delete phase. If you see over two hours in this phase,
  this APAR would apply. Similarly review the QUERY VOLUME output
  to see if there is one volume that has significantly more to
  reclaim than other tape volumes. If there is, then that volume
  would be left by itself during the reclaim phase and the slow
  down would be evident.
  
  Additionally, instrumentation tracing shows only one of these
  threads (SdReplicateCntr) active at a time during the reclaim
  phase of the process. If the slow down is during the delete
  phase, you can tell the delete processing is done in servermon
  when the SHOW THREADS output no longer displays the
  LocalProtectDispatchAndWaitDeletes function. If this lasts for
  over two hours this applies. Likewise on the ingest initial
  analysis side of things you can look for SdReplicateCntr after
  the delete phase and see how long that lasts in servermon
  output. If it is over two hours, this APAR likely applies.
  
  Initial Impact: Medium
  
  Additional Keywords: prot protect loc local del delete deletion
  recl reclaim reclamation perf performance slow slowdown LTO tape
  

Local fix

• Update the value for LOCALPROTCNTRCONS to a minimum value of 8
  if container-copy pool's PROTECTPRocess is 4 or less.  If
  PROTECTPRocess is greater than 4, update LOCALPROTCNTRCONS to
  twice the value of PROTECTPRocess.
  
  You can identify what the PROTECTPRocess setting for the
  container-copy pool by issuing the following command:
  
  QUERY STGPOOL <container-copy pool> FORMAT=DETAILED
  
  You can update the LOCALPROTCNTRCONS value with the following
  command:
  
  SETOPT LOCALPROTCNTRCONS <value>
  

Problem summary

• ****************************************************************
  * USERS AFFECTED:                                              *
  * All IBM Spectrum Protect server users of PROTECT STGPOOL     *
  * with TYPE=LOCAL                                              *
  ****************************************************************
  * PROBLEM DESCRIPTION:                                         *
  * See ERROR DESCRIPTION.                                       *
  ****************************************************************
  * RECOMMENDATION:                                              *
  * Apply fixing level when available. This problem is currently *
  * projected to be fixed in levels 8.1.7 and 8.1.1.300. Note    *
  * that this is subject to change at the discretion of IBM.     *
  ****************************************************************
  

Problem conclusion

• This problem was fixed.
  Affected platforms for reported release:  AIX, Linux, and
  Windows.
  Platforms fixed:  AIX, Linux, and Windows.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  TSM SERVER

• Fixed component ID

  5698ISMSV

Applicable component levels

• R81A PSY

     UP

• R81L PSY

     UP

• R81W PSY

     UP