IT20858: MULTISTREAM BACKUP SESSIONS MAY HANG WHEN SAVING DATA INTO DIRECTORY- OR CLOUD-CONTAINER STORAGE POOLS

8.1.3.000-IBM-SPSRV-AIX
8.1.3.000-IBM-SPSRV-Linuxppc64le
8.1.3.000-IBM-SPSRV-Linuxs390x
8.1.3.000-IBM-SPSRV-Linuxx86_64
8.1.3.000-IBM-SPSRV-WindowsX64
8.1.3.000-IBM-SPOC-AIX
8.1.3.000-IBM-SPOC-LinuxPPC64le
8.1.3.000-IBM-SPOC-Linuxs390x
8.1.3.000-IBM-SPOC-Linuxx86_64
8.1.3.000-IBM-SPOC-WindowsX64
8.1.3.000-IBM-SPCMS-Linuxx86_64
8.1.3.000-IBM-SPCMS-WindowsI32
8.1.3.000-IBM-SPCMS-WindowsX64
7.1.8.000-TIV-TSMSRV-AIX
7.1.8.000-TIV-TSMSRV-HP-UX
7.1.8.000-TIV-TSMSRV-Linuxppc64
7.1.8.000-TIV-TSMSRV-Linuxs390x
7.1.8.000-TIV-TSMSRV-Linuxx86_64
7.1.8.000-TIV-TSMSRV-SolarisSPARC
7.1.8.000-TIV-TSMSRV-WIN
7.1.8.000-TIV-TSMOC-AIX
7.1.8.000-TIV-TSMOC-Linuxx64
7.1.8.000-TIV-TSMOC-Windows
7.1.8.000-TIV-TSMOC-LinuxPPC64
7.1.8.000-TIV-TSMOC-LinuxPPC64le
7.1.8.000-TIV-TSMOC-LinuxS390
8.1.1.100-IBM-SPSRV-AIX
8.1.1.100-IBM-SPSRV-Linuxppc64le
8.1.1.100-IBM-SPSRV-Linuxs390x
8.1.1.100-IBM-SPSRV-Linuxx86_64
8.1.1.100-IBM-SPSRV-WindowsX64
IBM Spectrum Protect Server V8.1 Fix Pack 3 (V8.1.3) Downloads
IBM Spectrum Protect Server V8.1.1.X interim fix downloads

APAR status

• Closed as program error.

Error description

• Multistream backup sessions (for example backing up a Microsoft
  SQL DB with  IBM Spectrum Protect for Database ? Data Protection
   for Microsoft SQL Server using multiple stripes but the same
  could happen on DB2, SAP HANA, etc etc) may hang on the IBM
  Spectrum Protect Server if the data is stored into directory- or
   cloud-container storage pools. No error messages are reported
  on the server.
  
  
  
  Customer/L2 Diagnostics:
  The issue is caused by a wait for a lock on a new chunk being
  created that
  exceeds some internal timeout.
  It is possible to verify the issue by checking the output of
  the SHOW LOCKS command:
  
  Example:
  LockDesc: Type=20006(SD Pool SHA1 Value), NameSpace=5,
  SummMode=xLock, Key='C42B54A5D628B9B14403414848B486CD8BFBCCDF'
    Holder: (sddedup.c:1522 Thread 2257714) Tsn=0:2815687442,
  Mode=xLock
    Waiter: (sddedup.c:1573 Thread 243) Tsn=0:2815687602,
  Mode=sLock
    Waiter: (sddedup.c:1573 Thread 2257664) Tsn=0:2815687629,
  Mode=sLock
    Waiter: (sddedup.c:1573 Thread 2257662) Tsn=0:2815687647,
  Mode=sLock
  
  A subsequent check for the SHOW RESQ output will show how long
  the waiters have been waiting on the lock type 20006.
  A significant amount of time in the SHOW RESQ output (on the
  order of a high number of minutes) would be a good indication of
  this issue:
  
  ANS8000I Server command: 'SHOW RESQ'.
  RESOURCETimeout is 60 minutes.
  Resource list is unlocked.
  ==========================================================
  This is a local waiter.
  status=resWaiting waitTime=10 minutes
  waiter Type=unknown (0) timeout value=60 minutes
  txnSeqNo=0:2815687647
  resourceName=(C42B54A5D628B9B14403414848B486CD8BFBCCDF)
  lengthLen=40
  type=20006 and nameSpace=5
  waiter thead id is 2257662 condition=-577580872
   mutex=287340392 abortFunc=10d7fce0
  ==========================================================
  
  
  IBM Spectrum Protect Server Versions Affected: All IBM Spectrum
  Protect Server versions on all platforms
  
  Initial Impact: Medium
  
  Additional Keywords: hang stripes backup container sd
  deduplication mutiple sessions inline
  

Local fix

• Turn off inflight chunk checking on the server:
  
  setopt CheckInFlightChunks False
  
  Note:  As a side effect, duplicate data could potentially be
  stored  in the storage pool leading to higher storage pool
  utilization.
  This is something that cannot be quantified or predicted but
  depends on the nature of the data to be saved. It will only
  happen if the exact same base chunk is being newly created from
  multiple sessions simultaneously and this can occur only in very
  rare circumstances.
  

Problem summary

• ****************************************************************
  * USERS AFFECTED:                                              *
  * All IBM Spectrum Protect server users.                       *
  ****************************************************************
  * PROBLEM DESCRIPTION:                                         *
  * See error description.                                       *
  ****************************************************************
  * RECOMMENDATION:                                              *
  * Apply fixing level when available. This problem is currently *
  * projected to be fixed in level 8.1.3. Note that this is      *
  * subject to change at the discretion of IBM.                  *
  ****************************************************************
  

Problem conclusion

• This problem was fixed.
  Affected platforms for reported release:  AIX, Solaris, Linux,
  and Windows.
  Platforms fixed:  AIX, Linux, and Windows.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  TSM SERVER

• Fixed component ID

  5698ISMSV

Applicable component levels