IV50718: PERFORMANCE ISSUES WITH POLLER WHEN POLLING HIGH NUMBER OF ENTITIES

IBM Tivoli Network Manager IP 3.9.0 Fix Pack 4, 3.9.0-TIV-ITNMIP-FP0004
IBM Tivoli Network Manager IP Edition 3.9.0 Fix Pack 5, 3.9.0-TIV-ITNMIP-FP0005

APAR status

• Closed as program error.

Error description

• v3.9- FP2(also applicable for FP1 & FP3) - Solaris, but
  applicable for other platforms at higher monitoring load.
  
  When poller is configured to do ICMP polling of around 10k
  entities, poller falls behind with below message in
  ncp_poller.SnmpPoller.<DOMAIN>.trace:
  
  <snip>
  2013-10-02 11:12:43 SEVERE   schedule.cc(417): About to submit
  batch, but there aren't any threads available
  2013-10-02 11:12:45 SEVERE   schedule.cc(417): About to submit
  batch, but there aren't any threads available
  2013-10-02 11:12:46 SEVERE   schedule.cc(417): About to submit
  batch, but there aren't any threads available
  </snip>
  
  At this point - poller process memory size starts to increase
  and eventually it fails with Out-of-memory error.
  
  When the number of monitors are reduced, say 8k and recycle
  policy(disable followed by enabling policy) or restart Poller
  would address it.
  

Local fix

Problem summary

• ****************************************************************
  * USERS AFFECTED:                                              *
  * ITNM 3.9 FP2 (also applicable for FP1 & FP3) users.          *
  * In particular those operating on Solaris - however it is     *
  * applicable for other platforms at higher monitoring load.    *
  ****************************************************************
  * PROBLEM DESCRIPTION:                                         *
  * When poller is configured to do ICMP polling of around 10k   *
  * entities, the poller falls behind with below message in      *
  *                                                              *
  * ncp_poller.SnmpPoller.<DOMAIN>.trace:                        *
  *                                                              *
  * <snip>                                                       *
  * 2013-10-02 11:12:43 SEVERE   schedule.cc(417): About to      *
  * submit batch, but there aren't any threads available         *
  * 2013-10-02 11:12:45 SEVERE   schedule.cc(417): About to      *
  * submit batch, but there aren't any threads available         *
  * 2013-10-02 11:12:46 SEVERE   schedule.cc(417): About to      *
  * submit batch, but there aren't any threads available         *
  * </snip>                                                      *
  *                                                              *
  * At this point - poller process memory size starts to         *
  * increase and eventually it fails with Out-of-memory error.   *
  ****************************************************************
  * RECOMMENDATION:                                              *
  * Upgrade to                                                   *
  * Interim Fix 3.9.0.3-TIV-ITNMIP-IF0001                        *
  * and/or 3.9.0-ITNMIP-FP0004                                   *
  ****************************************************************
  

Problem conclusion

• An inefficiency in the Poller software was discovered which was
  causing degradation of performance in particular on the Solaris
  platform during some performance profiling as part of this
  investigation.
  
  The offending code has been addressed and will not now be
  executed unless the debug level is set to max.
  

Temporary fix

• Reduce the number of monitors to approx 8k and recycle policy
  (disable followed by enabling policy) or restart Poller would
  address it.
  

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  NC/PRECISIONIP

• Fixed component ID

  5724O52RC

Applicable component levels

• R390 PSN

     UP

• R390 PSY

     UP

• R410 PSN

     UP

• R410 PSY

     UP