IC72633: SWITCH AND SUBSYSTEM PERFORMANCE MONITORS FAIL DUE TO EXCESSIVE MEMORY CONSUMPTION.

4.2.1 Interim Fix 1 (November 2011) for Tivoli Storage Productivity Center

APAR status

• Closed as program error.

Error description

• Performance monitor jobs fail due to OutOfMemory conditions as a
  result of a defect in the monitor code.  Jobs fail, logging the
  following series of messages:
  HWNPM2123I Performance data for timestamp <timestamp> was
  collected and processed successfully. <number> performance data
  records were inserted into the database.
  HWNPM2026I The performance monitors primary process has failed
  unexpectedly. Attempting to recover from the failure.
  HWNPM2029I Successfully recovered from the performance monitor
  failure.
  To further diagnose the cause of these errors, review the
  msgTPCDeviceServer.log for the following type of OutOfMemory
  errors:
  com.ibm.tpc.disk.common.util.thread.DiskManagerRTException:
  com.ibm.tpc.disk.common.util.thread.DiskManagerExecException:
  java.lang.OutOfMemoryError at
  com.ibm.tpc.disk.common.CIM.CIMClientWrapper.executeSvcCall(CIMC
  lientWrapper.java:656) at
  com.ibm.tpc.disk.common.CIM.CIMClientWrapper.executeSvcCall(CIMC
  lientWrapper.java:362) at
  com.ibm.tpc.disk.common.CIM.CIMClientWrapper.enumerateInstances(
  CIMClientWrapper.java:1265) at
  com.ibm.tpc.discovery.cimom.CIMScanner.process(CIMScanner.java:3
  00) at
  com.ibm.tpc.infrastructure.threads.TPCThread.run(TPCThread.java:
  257)
  Caused by:
  com.ibm.tpc.disk.common.util.thread.DiskManagerExecException:
  java.lang.OutOfMemoryError at
  com.ibm.tpc.disk.common.util.thread.DiskThreadManager.executeWit
  hTimeLimit(DiskThreadManager.java:118) at
  com.ibm.tpc.disk.common.CIM.CIMClientWrapper.executeSvcCall(CIMC
  lientWrapper.java:475)
   ... 4 more
  Caused by: java.lang.OutOfMemoryError at
  org.sblim.wbem.http.io.TPCInputStream.buffer(TPCInputStream.java
  :53) at
  org.sblim.wbem.http.io.TPCInputStream.read(TPCInputStream.java:6
  1) at
  org.sblim.wbem.http.io.TPCInputStream.read(TPCInputStream.java:7
  6) at java.io.FilterInputStream.read(FilterInputStream.java:113)
  at
  sun.nio.cs.StreamDecoder$ConverterSD.implRead(StreamDecoder.java
  :354) at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:252)
  at java.io.InputStreamReader.read(InputStreamReader.java:212) at
  org.sblim.wbem.xml.parser.XMLPullParser.getChar(XMLPullParser.ja
  va:748) at
  org.sblim.wbem.xml.parser.XMLPullParser.getNextCharCheckingEOF(X
  MLPullParser.java:697) at
  org.sblim.wbem.xml.parser.XMLPullParser.next(XMLPullParser.java:
  170) at
  org.sblim.wbem.client.CIMEnumerationImpl.next(CIMEnumerationImpl
  .java:329) at
  org.sblim.wbem.client.CIMEnumerationImpl.<init>(CIMEnumerationIm
  pl.java:107) at
  org.sblim.wbem.client.CIMClientXML.getEnumeration(CIMClientXML.j
  ava:1870) at
  org.sblim.wbem.client.CIMClientXML.enumerateInstances(CIMClientX
  ML.java:573) at
  org.sblim.wbem.client.CIMClient.enumerateInstances(CIMClient.jav
  a:617) at
  org.sblim.wbem.client.CIMClient.enumerateInstances(CIMClient.jav
  a:524) at sun.reflect.GeneratedMethodAccessor72.invoke(Unknown
  Source) at
  sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethod
  AccessorImpl.java:43) at
  java.lang.reflect.Method.invoke(Method.java:618) at
  com.ibm.tpc.disk.common.util.thread.DiskMgrThread.run(DiskMgrThr
  ead.java:327)   com.ibm.tpc.discovery.cimom.CIMScanner process()
  Heap Dump files will be created in
  <TPC_Install_Dir>\device\apps\was\profiles\deviceServer
  RECREATE STEPS:
  Run a large number of performance monitor jobs in TPC 4.1.1 and
  observe errors.
  ________________________________________________________________
  DB2 Version used for Server:    N/A
  The defect is against component:   5648HWN01
  Server/Manager build/release (TPC):   4.1.1.66
  Agent build/release (TPC):    N/A
  Server/Manager (OS):     AIX 5.3
  Agent (OS):       N/A
  ________________________________________________________________
  Problem as described by customer:  large numbers of Perf
  Monitors fail
  Initial customer impact (low/med/high):  med
  

Local fix

• None Known
  

Problem summary

• ****************************************************************
  * USERS AFFECTED: TPC 4.1.1 or 4.2.1 customer scheduling       *
  *                 switch performance data collection.          *
  ****************************************************************
  * PROBLEM DESCRIPTION: When many switches are collecting       *
  *                      performance data, the                   *
  *                      OutOfMemoryError might be seen.  This   *
  *                      will sometimes cause the performance    *
  *                      data collection of other devices from   *
  *                      stopping.                               *
  ****************************************************************
  * RECOMMENDATION:                                              *
  ****************************************************************
  -
  

Problem conclusion

• This was fixed by improving and minimizing resource usage.
  The fix for this APAR is targeted for the following maintenance
  packages:
  
  | fix pack | 4.1.1-TIV-TPC-FP0006 - target April 2011
  | fix pack | 4.2.1-TIV-TPC-FP0002 - target February 2011
  
  http://www-01.ibm.com/support/docview.wss?&uid=swg21320822
  
  The target dates for future fix packs do not represent a formal
  commitment by IBM. The dates are subject to change without
  notice.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  MULTIPLE DEVICE

• Fixed component ID

  5648HWN01

Applicable component levels

• R411 PSY

     UP