IT22687: TIERING PROCESS MAY HANG WHEN STGRULE DURATION IS EXCEEDED

8.1.4.000-IBM-SPSRV-Linuxx86_64
8.1.4.000-IBM-SPSRV-WindowsX64
8.1.4.000-IBM-SPSRV-Linuxs390x
8.1.4.000-IBM-SPSRV-Linuxppc64le
8.1.4.000-IBM-SPSRV-AIX
IBM Spectrum Protect Server V8.1 Fix Pack 4 (V8.1.4) Downloads

APAR status

• Closed as program error.

Error description

• A IBM Spectrum Protect server tiering process may hang after the
  duration for
  the process has been exceeded. The tiering process cannot be
  cancelled.
  
  This condition can be exposed by a low MAXPROCESS value and / or
  slow network
  connection to the cloud.
  
  Customer/L2 Diagnostics:
  Looking at SHOW THREADS output will show the following
  condition.
  
  The parent thread is waiting for a condition on the queue to
  empty:
  Thread 892, Parent 891: ScParentTieringThread, Storage 435355,
  AllocCnt 1483 HighWaterAmt 474420
   tid=140132985759488, ptid=140136439260928, det=1, zomb=0,
  join=0, result=0, sess=0, procToken=2, sessToken=16
   lwp=19678
   Stack trace:
     0x007f7442a470af
  /lib64/libpthread.so.0(pthread_cond_wait+0xbf)
     0x0000000165b32b
  /opt/tivoli/tsm/server/bin/dsmserv(pkWaitConditionTracked+0xf8)
     0x0000000159fcfd
  /opt/tivoli/tsm/server/bin/dsmserv(CountedQueueWait+0x70)
     0x0000000159f48b
  /opt/tivoli/tsm/server/bin/dsmserv(WaitEmptyVarQueue+0xb6)
     0x00000000ffdc46
  /opt/tivoli/tsm/server/bin/dsmserv(WaitForTierStgpool+0x174)
     0x00000001000fbf
  /opt/tivoli/tsm/server/bin/dsmserv(ScParentTieringThread+0x808)
     0x00000001659223
  /opt/tivoli/tsm/server/bin/dsmserv(StartThread+0xc7)
     0x007f7442a42734 /lib64/libpthread.so.0(+0x8734)
     0x007f743e781d3d /lib64/libc.so.6(clone+0x6d)
    Awaiting cond newQueue->isEmpty (0x0x553e3f0), using mutex
  newQueue->mutex (0x0x7f73b0007a58), at queue.c(1743)
  
  The child thread has no stack trace and is a zombie (zomb=1),
  waiting to be joined by the parent:
  Thread 900, Parent 894: ScTierBatchesThread, Storage 286737916,
  AllocCnt 5769487 HighWaterAmt 295819773
   tid=140132986812160, ptid=140133001496320, det=0, zomb=1,
  join=0, result=3258, sess=0, procToken=0, sessToken=0
   lwp=19777
  
  Tivoli Storage Manager Versions Affected:
  IBM Spectrum Protect version 8.1.3 on all supported platforms.
  
  Initial Impact:
  High
  
  Additional Keywords:
  TSM, tiering, hang
  

Local fix

• Restart the server and increase the STGRULE DURATION and
  MAXPROCESS values or use a DURATION of NOLimit.
  

Problem summary

• ****************************************************************
  * USERS AFFECTED:                                              *
  * All IBM Spectrum Protect server users.                       *
  ****************************************************************
  * PROBLEM DESCRIPTION:                                         *
  * See error description.                                       *
  ****************************************************************
  * RECOMMENDATION:                                              *
  * Apply fixing level when available. This problem is currently *
  * projected to be fixed in level 8.1.4. Note that this is      *
  * subject to change at the discretion of IBM.                  *
  ****************************************************************
  

Problem conclusion

• This problem was fixed.
  Affected platforms for reported release: AIX, Linux, and
  Windows.
  Platforms fixed: AIX, Linux, and Windows.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  TSM SERVER

• Fixed component ID

  5698ISMSV

Applicable component levels

• R81A PSY

     UP

• R81L PSY

     UP

• R81W PSY

     UP