IV01053: DEPLOYMENT COMMAND SOMETIMES HANGS ON SOLARIS

IBM Tivoli Monitoring 6.2.2 Fix Pack 8 (6.2.2-TIV-ITM-FP0008)
IBM Tivoli Monitoring 6.2.2 Fix Pack 9 (6.2.2-TIV-ITM-FP0009)
IBM Tivoli Monitoring 6.2.2 Fix Pack 6 (6.2.2-TIV-ITM-FP0006)

APAR status

• Closed as program error.

Error description

• Occasionally (1 time out of 50) the execution of a deployment
  command
  (start, stop, configure, addSystem) results in the suspension of
  deployment processing on the endpoint machine because the mutex
  queue_mutex is left in a indeterminate state as a result of the
  Deploy Probe forking the OS Agent process.
  
  RECREATE INSTRUCTIONS:
  This condition rarely happens, and when it does it is only
  limited to
  Solaris OS endpoints.  The problem is due to the time window
  created by
  the presence of code executed between the fork of the OS Agent
  to create
  a child process and the actual spawning of a process by deploy.
  The spawned deploy process is the Operating System Shell
  (bin/sh).
  The current time window created between the fork of the OS Agent
  and the
  spawning of the child process is slowed by the processing to
  create a
  unique RAS1 log file name for the child process.
  Since forking a child process does not copy the current state of
  mutexes,
  it is possible for any of the mutexes used by Remote Deploy to
  have an
  indeterminate state,this includes the queue_mutex that is
  responsible
  for handling incoming deployment requests.
  So as far as recreate instructions, this is the procedure that
  may or
  may not result in the reported failure.
  
  1. Install ITM 622 FP4 on TEMS
  2. Install ITM 622 FP4 OS Agent on Solaris endpoint.
  3. Install ITM 622 FP4 UL Agent on same endpoint.
  4. On TEMS, log into CLI
       tacmd login -s localhost -u root -t 1440
  5. On TEMS issue command to stop UL Agent
       tacmd agent stop -n mymachine:UL
       tacmd agent start -n mymachine:UL
  6. Repeat step 5 until failure is seen.
  System Configuration:  Sun Microsystems  sun4v
  Memory size: 24576 Megabytes
  System Peripherals (Software Nodes):
  

Local fix

• Manually restart OS Agent to clear condition when it is present.
  

Problem summary

• OS Agent occasionally stops processing remote commands and
  appears to hang.
  
  Occasionally when executing a remote command that performs,
  stop, start, restart, install, uninstall, set configuration, or
  remove instance the OS Agent will no longer process remote
  commands, and will appear to hang.
  

Problem conclusion

• Changed the thread processing algorithm that handles incoming
  remote requests such that the queuing of the request is
  independent of its processing.  This will eliminate potential
  blocking that may occur for requests that must be placed on the
  processing queue.  Additionally removed processing between the
  fork and execvp calls that process log file renaming since the
  string processing involved in this processing extensively uses
  mutexes which do not retain their value in the spawned child
  process.
  
  
  The fix for this APAR is contained in the following maintenance
  packages:
  
    | fix pack | 6.2.2-TIV-ITM-FP0006
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  TEMS

• Fixed component ID

  5724C04MS

Applicable component levels

• R622 PSY

     UP