IV81845: THE IFSTAT DAEMON BECOMES UNRESPONSIVE ON AIX.

IBM Tivoli Monitoring: Unix(R) OS Agent 6.3.0.6-TIV-ITM_UNIX-IF0001
IBM Tivoli Monitoring: Unix(R) OS Agent 6.3.0.4-TIV-ITM_UNIX-IF0004
IBM Tivoli Monitoring 6.3.0 Fix Pack 7 (6.3.0-TIV-ITM-FP0007)
IBM Tivoli Monitoring: Unix(R) OS Agent 6.3.0.5-TIV-ITM_UNIX-IF0005
IBM Tivoli Monitoring: Unix(R) OS Agent 6.3.0.6-TIV-ITM_UNIX-IF0003
IBM Tivoli Monitoring: Unix(R) OS Agent 6.3.0.6-TIV-ITM_UNIX-IF0004

APAR status

• Closed as program error.

Error description

• The ifstat daemon becomes unresponsive due to an incomplete
  response of the api perfstat_netinterface.
  
  The UNIX OS Agent is composed by 7 processes: kuxagent,
  stat_daemon, ifstat, kuxdstat (iostat on AIX), kux_vmstat, and
  nfs_stat and (mount_stat - 6.23 onwards)
  
  The ifstat process is spawned by the stat_daemon and is
  responsible for collecting every 30s the metrics for the
  following attribute group: UNIXNET
  
  Details:
  Gather logs with the following trace in lz.ini
  KBB_RAS1=ERROR (UNIT:KUX ALL) (UNIT:CPU ALL) (UNIT:stat_daemon
  ALL) (UNIT:ifstat ALL)
  DEBUG_PURPOSE_ONLY=TRUE
  
  The problem occurs because:
  1) the first call to perfstat_netinterface replies with 7
  interfaces
  2) the first call to perfstat_netinterface (to get the data)
  replies with ONLY 2 interfaces.
  
  So there is a clear mistmatch, which causes the issue.
  
  NOTE: The numbers may be different in each environment. The
  above example was recorded in the problem.
  
  Log entries: <hostname>_ux_ifstat_<timestamp>-01.log
  
  (56CF1C43.0001-1:ifstat-bsd.cpp,476,"main") 7 interfaces found
  
  between TWO messages of the same type, you can count the
  occurrences of the following message
  (56CF1C43.0025-1:ifstat-bsd.cpp,621,"main") name:
  
  In this case it was just 2 instead of 7.
  

Local fix

Problem summary

• Problem Summary: Monitoring Agent for UNIX OS becomes
  unresponsive on AIX due to an incomplete response of the AIX api
  perfstat_netinterface
  
  Problem Conclusion: The ifstat daemon has been fixed to manage
  the unexpected behavior of the AIX api perfstat_netinterface
  

Problem conclusion

• The fix for this APAR is included in the following maintenance
  vehicle:
  
     | fix pack | 6.3.0-TIV-ITM-FP0007 |
     | interim fix | 6.3.0.4-TIV-ITM_UNIX-IF0004 |
     | interim fix | 6.3.0.5-TIV-ITM_UNIX-IF0005 |
     | interim fix | 6.3.0.5-TIV-ITM_UNIX-IF0006 |
     | interim fix | 6.3.0.5-TIV-ITM_UNIX-IF0007 |
     | interim fix | 6.3.0.6-TIV-ITM_UNIX-IF0001 |
  
  Note: Search the IBM technical support web site for maintenance
  package availability.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  ITM AGENT UNIX

• Fixed component ID

  5724C040U

Applicable component levels