IT23030: CONTAINER POOL SPACE NOT FREEING AS EXPECTED, ANR9999D_1019972953 INVALID PARAMETER(S): 0 SEEN IN ACTLOG

8.1.1.300-IBM-SPSRV-WindowsX64
8.1.1.300-IBM-SPSRV-Linuxx86_64
8.1.1.300-IBM-SPSRV-Linuxs390x
8.1.1.300-IBM-SPSRV-Linuxppc64le
8.1.1.300-IBM-SPSRV-AIX
7.1.9.000-TIV-TSMCMS-Windows64
7.1.9.000-TIV-TSMCMS-Windows32
7.1.9.000-TIV-TSMCMS-Linuxx64
7.1.9.000-TIV-TSMOC-LinuxS390
7.1.9.000-TIV-TSMOC-LinuxPPC64
7.1.9.000-TIV-TSMOC-Windows
7.1.9.000-TIV-TSMOC-Linuxx64
7.1.9.000-TIV-TSMOC-AIX
7.1.9.000-TIV-TSMSRV-WIN
7.1.9.000-TIV-TSMSRV-SolarisSPARC
7.1.9.000-TIV-TSMSRV-Linuxx86_64
7.1.9.000-TIV-TSMSRV-Linuxs390x
7.1.9.000-TIV-TSMSRV-Linuxppc64
7.1.9.000-TIV-TSMSRV-HP-UX
7.1.9.000-TIV-TSMSRV-AIX
8.1.5.000-IBM-SPCMS-WindowsX64
8.1.5.000-IBM-SPCMS-WindowsI32
8.1.5.000-IBM-SPCMS-Linuxx86_64
8.1.5.000-IBM-SPOC-WindowsX64
8.1.5.000-IBM-SPOC-Linuxx86_64
8.1.5.000-IBM-SPOC-Linuxs390x
8.1.5.000-IBM-SPOC-LinuxPPC64le
8.1.5.000-IBM-SPOC-AIX
8.1.5.000-IBM-SPSRV-WindowsX64
8.1.5.000-IBM-SPSRV-Linuxx86_64
8.1.5.000-IBM-SPSRV-Linuxs390x
8.1.5.000-IBM-SPSRV-Linuxppc64le
8.1.5.000-IBM-SPSRV-AIX
IBM Spectrum Protect Server V8.1 Fix Pack 5 (V8.1.5) Downloads
IBM Spectrum Protect Server V7.1 Fix Pack 9 (7.1.9.000) Downloads
IBM Spectrum Protect Server V8.1.1.X interim fix downloads

APAR status

• Closed as program error.

Error description

• Error Description:
  
  If data has been been removed from a directory or cloud
  container pool (example : deleted a node's filespace) , the
  extents are deleted via a background thread to free the space
  within the container pool.
  During the removal of extents, the background deletion thread
  may encounter extents that have an invalid container id and
  could result in the thread being blocked from performing any
  further extent removal.
  This could lead to increased storage pool utilization as space
  within the container pool cannot be freed, although the data
  itself will show as being removed at the node level (QUERY OCC
  command).
  
  The following errors will be seen within the activity log :
  
  09/27/2017 20:59:06      ANR9999D_1019972953
  SdSetGlobalListContainerNeedsRefresh(-
                            sdcntr.c:4982) Thread<157>: Invalid
  parameter(s): 0.
  09/27/2017 20:59:06      ANR9999D Thread<157> issued message
  9999 from:
  09/27/2017 20:59:06      ANR9999D Thread<157>  0x00000001064345
  OutDiagToCons
  09/27/2017 20:59:06      ANR9999D Thread<157>  0x0000000106757c
  outDiagfExt
  09/27/2017 20:59:06      ANR9999D Thread<157>  0x00000000cba920
                            SdSetGlobalListContainerNeedsRefresh
  09/27/2017 20:59:06      ANR9999D Thread<157>  0x00000000ce1e16
                            SdChunkDelWorkerThread
  09/27/2017 20:59:06      ANR9999D Thread<157>  0x00000001096b75
  PcConsumerThread
  09/27/2017 20:59:06      ANR9999D Thread<157>  0x0000000111aed2
  StartThread
  09/27/2017 20:59:06      ANR9999D Thread<157>  0x007ffff76acdc5
  *UNKNOWN*
  09/27/2017 20:59:06      ANR9999D Thread<157>  0x007ffff385976d
  *UNKNOWN*
  
  It is also known that if a negative value for the container id
  is encountered,
  this will lead to the sever crashing and the following stack is
  seen :
  
  adsmdll!SdSetGlobalListContainerNeedsRefresh+0xb4
  adsmdll!SdChunkDelWorkerThread+0x438
  adsmdll!PcConsumerThread+0x141
  adsmdll!startThread+0x141
  msvcr120!_callthreadstartex+0x17
  msvcr120!_threadstartex+0x102
  kernel32!BaseThreadInitThunk+0x22
  ntdll!RtlUserThreadStart+0x34
  
  
  
  Customer/L2 Diagnostics (If Applicable)
  
  To locate invalid container id's, issue the following :
  
  db2 connect to tsmdb1
  
  Obtain the poolid for the affected storage pool :
  
  db2 "select poolid from tsmdb1.ss_pools where
  poolname=upper('YourDirPool')"
  
  Using the above pool id value, replace POOLID in the following
  statement :
  
  db2 db2 "SELECT cl.cntrid,count(*) FROM
  tsmdb1.sd_chunk_locations cl where not exists(select co.cntrid
  from tsmdb1.sd_containers co where co.cntrid=cl.cntrid) and
  cl.poolid=POOLID group by cl.cntrid FOR READ ONLY WITH UR"
  
  Tivoli Storage Manager Versions Affected:
  IBM Spectrum Protect versions 7.1.3 and and higher and 8.1 and
  higher on all platforms.
  
  Initial Impact: Low|Medium|High
  Medium
  
  Additional Keywords:
  TSM, container, ANR0522W, out of space, chunk, removal, full,
  crash
  

Local fix

• Contact IBM support for corrective steps.
  

Problem summary

• ****************************************************************
  * USERS AFFECTED:                                              *
  * All Tivoli Storage Manager and IBM Spectrum Protect server   *
  * users.                                                       *
  ****************************************************************
  * PROBLEM DESCRIPTION:                                         *
  * See error description.                                       *
  ****************************************************************
  * RECOMMENDATION:                                              *
  * Apply fixing level when available. This problem is currently *
  * projected to be fixed in levels 8.1.5 and 7.1.9. Note that   *
  * this is subject to change at the discretion of IBM.          *
  ****************************************************************
  

Problem conclusion

• This problem was fixed.
  Affected platforms for reported release: AIX, Solaris, Linux,
  and Windows.
  Platforms fixed: AIX, Solaris, Linux, and Windows.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  TSM SERVER

• Fixed component ID

  5698ISMSV

Applicable component levels

• R71A PSY

     UP

• R71L PSY

     UP

• R71S PSY

     UP

• R71W PSY

     UP

• R81A PSY

     UP

• R81L PSY

     UP

• R81W PSY

     UP