IT02590: CLIENT SESSIONS HUNG IN RUN STATE DUE TO SESSION IN MEDIAW

APAR status

• Closed as program error.

Error description

• Client session is hung in MediaW state.  New client and admin
  sessions hang in the Run state and in time the server can run
  out of sessions preventing new client/admin sessions.  These
  symptoms are the same as APAR IC99934, but the cause and
  identifiers are different.
  
  Tivoli Storage Manager Versions Affected: All V62, V63 and V7
  Servers
  
  Customer/L2 Diagnostics:
  Query Session will show growing number of client sessions in the
  run state:
   Sess       Comm.    Sess     Wait    Sess
   Number     Method   State    Time    Type
   -------    ------   ------  ------   -------
   1,336,400  Tcp/Ip   SendW    0 S     Node
   1,337,000  Tcp/Ip   MediaW 4.7 M     Node
   1,337,301  Tcp/Ip   Run      0 S     Node
   1,337,302  Tcp/Ip   Run      0 S     Node
   1,337,303  Tcp/Ip   Run      0 S     Node
   1,337,304  Tcp/Ip   Run      0 S     Node
   1,337,305  Tcp/Ip   Run      0 S     Node
   1,337,306  Tcp/Ip   Run      0 S     Admin
   ...
  
  Collect the servermonV6.pl script data during the time you get
  traces.
  Although this problem is difficult to trap with a trace, the
  following flags can be used:
  TRACE ENABLE INSTR AS MMS PVRNOIO
  
  Tracing with just INSTR and PVR seemed to help trigger the
  problem more frequently.
  
  The traces will show a latch deadlock.
  
  The deadlocked threads will both have the same stack.
  Example:
  Thread 3631144, Parent 1048: psSessionThread, Storage 7797835,
  AllocCnt 1621 HighWaterAmt 7867735
    tid=828, ptid=2218, det=1, zomb=0, join=0, result=0,
  sess=871784
     Awaiting cond latchP->sFree (0x110cdc6f0), using mutex
  PVRV->mutex (0x111b8c148), at latch.c(256)
     Stack trace:
       0x09000000004b9ab0 _cond_wait_global
       0x09000000004ba63c _cond_wait
       0x09000000004bb32c pthread_cond_wait
       0x00000001000079f4 pkWaitConditionTracked
       0x000000010068a51c SmNodeSession
       0x000000010053e39c HandleNodeSession
       0x0000000100535cb4 DoNodeGeneral
       0x0000000100530ce0 smExecuteSession
       0x000000010017827c psSessionThread
       0x000000010000c264 StartThread
  
  Trace data:
  06:56:18.143 [3631144][latch.c][238][AcquireLatchSpecific]:LATCH
  Acquire ATTEMPTED for latch
  12740150, mode 0, using mutex 11b8c008 by requestor
  pvrclass.c(3631144).
  06:56:18.143 [3631144][latch.c][294][AcquireLatchSpecific]:LATCH
  Acquire SUCCESSFUL for latch
  12740150, mode 0, using mutex 11b8c008 by requestor
  pvrclass.c(3631144).
  06:56:18.142 [3631155][latch.c][238][AcquireLatchSpecific]:LATCH
  Acquire ATTEMPTED for latch
  12750d90, mode 1, using mutex 11b8c008 by requestor
  pvrmp.c(938)(3631155).   ----> Another thread
  requesting for Xlatch blocked the 3631144's Slatch request
  
  Initial Impact: High
  Additional Keywords: Hang hung Login failed session client
  backups fail
  

Local fix

• Restart the Tivoli Storage Manager server
  

Problem summary

• ****************************************************************
  * USERS AFFECTED:                                              *
  * All Tivoli Storage Manager server users.                     *
  ****************************************************************
  * PROBLEM DESCRIPTION:                                         *
  * See ERROR DESCRIPTION.                                       *
  ****************************************************************
  * RECOMMENDATION:                                              *
  * Apply fixing level when available. This problem is currently *
  * projected to be fixed in levels 6.3.5 and 7.1.1. Note that   *
  * this is subject to change at the discretion of IBM.          *
  ****************************************************************
  

Problem conclusion

• This problem was fixed.
  Affected platforms:  AIX, HP-UX, Solaris, Linux, and Windows.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  TSM SERVER

• Fixed component ID

  5698ISMSV

Applicable component levels

• R63A PSY

     UP

• R63H PSY

     UP

• R63L PSY

     UP

• R63S PSY

     UP

• R63W PSY

     UP

• R71A PSY

     UP

• R71H PSY

     UP

• R71L PSY

     UP

• R71S PSY

     UP

• R71W PSY

     UP