IT26168: MESSAGE FABRIC SERVICE CAUSES AN OUT OF MEMORY ERROR

APAR status

• Closed as program error.

Error description

• Error Description
  
  MessageFabricServiceImpl has used occupies 374,049,712 (71.61%)
  bytes memory heap and has caused OOM error which kept bringing
  the node down.
  
  
  The heap dump analysis shows major suspect was message fabric
  
  
  The class
  "com.ibm.b2b.osgi.bundles.msgfabric.RequestReplySessionManager",
  loaded by "<system class loader>", occupies 374,049,712 (71.61%)
  bytes. The memory is accumulated in one instance of
  "java.util.concurrent.ConcurrentHashMap$Segment[]" loaded by
  "<system class loader>".
  
  
  Keywords
  
  com.ibm.b2b.osgi.bundles.msgfabric.RequestReplySessionManager
  
  java.util.concurrent.ConcurrentHashMap$Segment[]
  
  
  
  
  There were log messages like
  
  
  [1557496804124]java.nio.file.ClosedWatchServiceException
  
      at sun.nio.fs.AbstractPoller.invoke(AbstractPoller.java:229)
  
      at
  sun.nio.fs.AbstractPoller.register(AbstractPoller.java:141)
  
      at
  sun.nio.fs.WindowsWatchService.register(WindowsWatchService.java
  :86)
  
      at sun.nio.fs.WindowsPath.register(WindowsPath.java:888)
  
      at sun.nio.fs.AbstractPath.register(AbstractPath.java:117)
  
      at
  com.ibm.b2b.storage.service.adapter.StorageAdapterDirectoryWatch
  er.registerPathForWatching(StorageAdapterDirectoryWatcher.java:2
  59)
  
      at
  com.ibm.b2b.storage.service.adapter.StorageAdapterDirectoryWatch
  er.registerBaseFilePath(StorageAdapterDirectoryWatcher.java:223)
  
      at
  com.ibm.b2b.storage.service.adapter.StorageAdapterDirectoryWatch
  er.registerBucketBaseFilePath(StorageAdapterDirectoryWatcher.jav
  a:210)
  
      at
  com.ibm.b2b.osgi.bundles.megstorage.MEGStoreContextManager.getSt
  oreContext(MEGStoreContextManager.java:147)
  
      at
  com.ibm.b2b.osgi.bundles.megstorage.MEGStorageServiceImpl.getSto
  reContext(MEGStorageServiceImpl.java:400)
  
      at
  com.ibm.b2b.osgi.bundles.megstorage.MEGStorageServiceImpl.getDat
  a(MEGStorageServiceImpl.java:211)
  
      at
  com.ibm.b2b.services.businessdoc.StorageKeeper.getData(StorageKe
  eper.java:126)
  
      at
  com.ibm.b2b.osgi.bundles.msgfabric.BusinessDocCallback.extractDo
  c(BusinessDocCallback.java:604)
  
      at
  com.ibm.b2b.osgi.bundles.msgfabric.BusinessDocCallback.toProcess
  Data(BusinessDocCallback.java:317)
  

Local fix

• STRRTC - 573125
  DW / DW
  Circumvention :
  
  Please check the messagefabric.log if you have the following
  error repeatedly occurring in the log:
  [2018-07-27 18:21:16.151] ERROR BusinessDocCallback.onMessage
  caught throwable.
  [2018-07-27 18:21:16.151] ERROR [1532679676151]
  com.ibm.b2b.apiint.messagedefinitions.messages.xml.exchange.noti
  fication.NotifyErrorType incompatible with
  com.ibm.b2b.apiint.messagedefinitions.messages.xml.exchange.as4n
  otification.NotifyErrorMessage
  [2018-07-27 18:21:16.151] ERRORDTL
  [1532679676151]java.lang.ClassCastException:
  com.ibm.b2b.apiint.messagedefinitions.messages.xml.exchange.noti
  fication.NotifyErrorType incompatible with
  com.ibm.b2b.apiint.messagedefinitions.messages.xml.exchange.as4n
  otification.NotifyErrorMessage
  at
  com.ibm.b2b.osgi.bundles.msgfabric.BusinessDocCallback.toProcess
  Data(BusinessDocCallback.java:367)
  at
  com.ibm.b2b.osgi.bundles.msgfabric.ProcessLauncherCallback.onMes
  sage(ProcessLauncherCallback.java:142)
  at
  com.ibm.b2b.osgi.bundles.msgfabric.BusinessDocCallback.onMessage
  (BusinessDocCallback.java:157)
  at
  com.ibm.b2b.osgi.bundles.msgfabric.CompositeCallback.onMessage(C
  ompositeCallback.java:183)
  at
  com.ibm.b2b.osgi.bundles.msgfabric.AsyncMessageHandler.onMessage
  (AsyncMessageHandler.java:112)
  
  If yes, you need to:
  1. Identify the BDO Adapter that triggers the error.
  2. Disable the adapter and check the last message (invalid
  message) from the queue.
  3. Remove the message from the queue and restart the adapter.
  Ensure the error message disappears from the log.
  

Problem summary

• Users Affected:
  All
  
  Problem Description:
  The Message Fabric Service causes an out of memory error in the
  adapter container due to ClassCastException.
  
  Platforms Affected:
  All
  

Problem conclusion

• Resolution Summary:
  
  A code fix is provided.
  
  Fix for OOM is to make the unregister operation when there is a
  failure in Notification message:
  
  BusinessDocCallback.toProcessData
  
  
  Any faulty message processing which goes for loop back message
  handling need to unregister itself first before opening a new
  session to process.
  
  Delivered In:
  5020603_11
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  STR B2B INTEGRA

• Fixed component ID

  5725D0600

Applicable component levels