IC67615: UPDATE DRIVE COMMANDS DURING DRIVE POLLING CAN CAUSE PARTIAL SER VER HANG

IBM Tivoli Storage Manager V6.1 Fix Pack 4 (6.1.4) Server Downloads

APAR status

• Closed as program error.

Error description

• The Tivoli Storage Manager server can partially hang because of
  a deadlock issue if multiple UPDATE DRIVE commands are issued
  while drive polling is taking place.
  .
  The first symptoms will be that various administrative commands
  such as QUERY VOLUME and QUERY MOUNT will not return data.
  Client backups may also stop.  Eventually, the server can
  completely hang.
  .
  Customer/L2 Diagnostics:
  Collect several iterations of "SHOW THREADS" over a period
  of time.  Search for a SmAdminCommandThread thread that includes
  the "mmsCmdUpdateDrive" function.  If this same thread remains
  in a "_cond_wait_global" wait condition during subsequent "SHOW
  THREADS" iterations (over 30 minutes or longer), this problem
  likely applies.  For example;
  .
  Thread 12345, Parent 54321: SmAdminCommandThread, Storage 22494,
  AllocCnt 23 HighWaterAmt 38617
   tid=19ac9, ptid=194bc, det=0, zomb=0, join=1, result=0, sess=0
    Awaiting cond driveP->AckTerminatePoll (0x110947840),
    using mutex MMSV->mutex (0x111189138), at mmsdrive.c(4253)
    Stack trace:
      0x0900000000710d50 _cond_wait_global
      0x09000000007118dc _cond_wait
      0x09000000007125cc pthread_cond_wait
      0x00000001000076c4 pkWaitConditionTracked
      0x0000000100037a34 MmsUpdateDrive
      0x0000000100a54288 mmsCmdUpdateDrive
      0x0000000100ac6ef8 psCommand
      0x000000010016ea04 AdmCommandLocal
      0x000000010016d198 admCommand
      0x0000000100753e74 SmAdminCommandThread
      0x0000000100009504 StartThread
  .
  Initial Impact:
  High
  .
  Tivoli Storage Manager Versions Affected:
  All supported Tivoli Storage Manager servers.
  .
  Additional Keywords:
  TSM DSMSERV HANG HUNG PARTIAL COMMAND UPDATE DRIVE ZZ61
  FAILURE MUTEX MMSV->MUTEX PKWAITCONDITIONTRACKED
  .
  

Local fix

• Restart the server to clear the deadlock.
  

Problem summary

• ****************************************************************
  * USERS AFFECTED: All Tivoli Storage Manager servers using     *
  *                 tape drives.                                 *
  ****************************************************************
  * PROBLEM DESCRIPTION: See ERROR DESCRIPTION.                  *
  ****************************************************************
  * RECOMMENDATION: Apply fixing level when available. This      *
  *                 problem is currently projected to be fixed   *
  *                 in level 5.4.7, 5.5.5, 6.1.4, and 6.2.1.     *
  *                 Note that this is subject to change at the   *
  *                 discretion of IBM.                           *
  ****************************************************************
  *
  

Problem conclusion

• This problem was fixed.
  Affected platforms:  AIX, Sun Solaris, Linux, HP-UX, and
  Windows.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  TSM SERVER

• Fixed component ID

  5698ISMSV

Applicable component levels

• R54A PSY

     UP

• R54H PSY

     UP

• R54L PSY

     UP

• R54S PSY

     UP

• R54W PSY

     UP

• R55A PSY

     UP

• R55H PSY

     UP

• R55L PSY

     UP

• R55S PSY

     UP

• R55W PSY

     UP

• R61A PSY

     UP

• R61H PSY

     UP

• R61L PSY

     UP

• R61S PSY

     UP

• R61W PSY

     UP

• R62A PSY

     UP

• R62H PSY

     UP

• R62L PSY

     UP

• R62S PSY

     UP

• R62W PSY

     UP