IT22346: SERVER MAY HANG DURING HIGH WORKLOAD

7.1.7.400-TIV-TSMSRV-WIN
7.1.7.400-TIV-TSMSRV-SolarisSPARC
7.1.7.400-TIV-TSMSRV-Linuxx86_64
7.1.7.400-TIV-TSMSRV-Linuxs390x
7.1.7.400-TIV-TSMSRV-Linuxppc64
7.1.7.400-TIV-TSMSRV-HP-UX
7.1.7.400-TIV-TSMSRV-AIX
7.1.9.000-TIV-TSMCMS-Windows64
7.1.9.000-TIV-TSMCMS-Windows32
7.1.9.000-TIV-TSMCMS-Linuxx64
7.1.9.000-TIV-TSMOC-LinuxS390
7.1.9.000-TIV-TSMOC-LinuxPPC64
7.1.9.000-TIV-TSMOC-Windows
7.1.9.000-TIV-TSMOC-Linuxx64
7.1.9.000-TIV-TSMOC-AIX
7.1.9.000-TIV-TSMSRV-WIN
7.1.9.000-TIV-TSMSRV-SolarisSPARC
7.1.9.000-TIV-TSMSRV-Linuxx86_64
7.1.9.000-TIV-TSMSRV-Linuxs390x
7.1.9.000-TIV-TSMSRV-Linuxppc64
7.1.9.000-TIV-TSMSRV-HP-UX
7.1.9.000-TIV-TSMSRV-AIX
8.1.4.000-IBM-SPSRV-Linuxx86_64
8.1.4.000-IBM-SPSRV-WindowsX64
8.1.4.000-IBM-SPSRV-Linuxs390x
8.1.4.000-IBM-SPSRV-Linuxppc64le
8.1.4.000-IBM-SPSRV-AIX
8.1.1.100-IBM-SPSRV-WindowsX64
8.1.1.100-IBM-SPSRV-Linuxx86_64
8.1.1.100-IBM-SPSRV-Linuxs390x
8.1.1.100-IBM-SPSRV-Linuxppc64le
8.1.1.100-IBM-SPSRV-AIX
IBM Spectrum Protect Server V8.1.1.X interim fix downloads
IBM Spectrum Protect Server V8.1 Fix Pack 4 (V8.1.4) Downloads
IBM Spectrum Protect Server V7.1 Fix Pack 9 (7.1.9.000) Downloads
IBM Spectrum Protect Server V7.1.7.X interim fix downloads

APAR status

• Closed as program error.

Error description

• In rare circumstances, when IBM Spectrum Protect server is under
  a high workload, server may hang indefinitely. An instance
  restart is needed to clear the hang.
  
  This can happen if following conditions are met:
  
  - A status monitor is turned on and is collecting data on a
  default interval of 15 minutes.
  - A DEFINE ASSOCIATION command is in progress for a node that is
  in the process of recording a summary record.
  
  Note: A server script would be most likely suspected to issue
  commands on a regular period.
  
  The symptom is also that Servermon Perl script will hang as well
  and not collect any data during the server hang. A procstack or
  a core dump needs to be obtained from dsmserv process, which
  will contain a call stack similar to following:
  
  pth_cond._cond_wait_global(??, ??, ??) at 0x90000000051cd50
  pth_cond._cond_wait(??, ??, ??) at 0x90000000051d8d8
  pth_cond.pthread_cond_wait(??, ??) at 0x90000000051e5c8
  pkmon.pkWaitConditionTracked(??, ??, ??, ??, ??) at 0x100008f10
  tmlock.WaitForLock(??, ??, ??, ??, ??, ??, ??, ??) at
  0x10027f680
  tmlock.tmLockTracked(??, ??, ??, ??, ??, ??, ??, ??) at
  0x10027dd0c
  admutil.AdmLockNode(??, ??, ??) at 0x1000e21c0
  admnode.admGetNodeExtAttrs(??, ??, ??, ??) at 0x1002f2f1c
  monstats.monLogClientSchedule(??, ??, ??, ??, ??, ??, ??, ??) at
  0x1000b0834
  csutil.csMonitorClientSchedules() at 0x1002bb0d0
  monstats.StatusMonitorGridsThread(??) at 0x1000a94dc
  pkthread.StartThread(0x0) at 0x10000da90
  
  The admGetNodeExtAttrs() method must be present for this APAR to
  apply.
  
  
  IBM Spectrum Protect Versions Affected:
  IBM Spectrum Protect server on all platforms
  
  
  Initial Impact:
  Medium
  
  
  Additional Keywords:
  Spectrum Protect; TSM; server; hang; admGetNodeExtAttrs;
  monLogClientSchedule; csMonitorClientSchedules;
  StatusMonitorGridsThread
  

Local fix

• 1) Disable the status monitor during the backup window or
  increase the status monitor refreshinterval.
  2) Do not issue the DEFINE ASSOCIATION commands for a node
  during the time frame of that node backup operation.
  

Problem summary

• ****************************************************************
  * USERS AFFECTED:                                              *
  * All IBM Tivoli Storage Manager and IBM Spectrum Protect      *
  * server users.                                                *
  ****************************************************************
  * PROBLEM DESCRIPTION:                                         *
  * See error description.                                       *
  ****************************************************************
  * RECOMMENDATION:                                              *
  * Apply fixing level when available. This problem is currently *
  * projected to be fixed in levels 7.1.9 and 8.1.4. Note that   *
  * this is subject to change at the discretion of IBM.          *
  ****************************************************************
  

Problem conclusion

• This problem was fixed.
  Affected platforms for reported release:  AIX, HP-UX, Solaris,
  Linux, and Windows.
  Platforms fixed:  AIX, HP-UX, Solaris, Linux, and Windows.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  TSM SERVER

• Fixed component ID

  5698ISMSV

Applicable component levels

• R71A PSY

     UP

• R71H PSY

     UP

• R71L PSY

     UP

• R71S PSY

     UP

• R71W PSY

     UP

• R81A PSY

     UP

• R81L PSY

     UP

• R81W PSY

     UP