PH12258: Client ObjectGrid instances can leak when they are not destroyed.

APAR status

• Closed as program error.

Error description

• System outages can occur when applications are coded to create
  many client ObjectGrid instances and not destroy them when
  done with them. An example symptom can be OutOfMemoryError
  exceptions occuring in your application that is a client to an
  eXtreme Scale data grid. Also when stopping client applications
  with many ObjectGrid instances for the same ObjectGrid, errors
  can occur on the containers side like those below:
  
  XSThreadPool W CWOBJ7853W: Detected a hung thread named
  "XIOPrimaryPool : 107" TID:434 BLOCKED. Executing since
  5/16/2019 01:02:09:699 +0200.
  Stack Trace:
  com.ibm.ws.xs.pubsub.publication.Publisher.dispatchException(Pub
  lisher.java:772)
  com.ibm.ws.xsspi.xio.actor.DispatchExceptionListenerRegistry.dis
  patchExceptionEncountered(DispatchExceptionListenerRegistry.java
  :249)
  com.ibm.ws.xsspi.xio.actor.DispatchExceptionListenerRegistry.rec
  eive(DispatchExceptionListenerRegistry.java:78)
  com.ibm.ws.xs.xio.actor.impl.XIOReferableImpl.dispatch(XIORefera
  bleImpl.java:110)
  com.ibm.ws.xsspi.xio.actor.XIORegistry.sendToTarget(XIORegistry.
  java:973)
  com.ibm.ws.xsspi.xio.dispatch.MessageDispatcher.tell(MessageDisp
  atcher.java:168)
  com.ibm.ws.xs.xio.actor.impl.ActorRefImpl.tell(ActorRefImpl.java
  :66)
  com.ibm.ws.xsspi.xio.dispatch.DispatchExceptionRunnable.sendErro
  r(DispatchExceptionRunnable.java:203)
  com.ibm.ws.xsspi.xio.dispatch.DispatchExceptionRunnable.run(Disp
  atchExceptionRunnable.java:159)
  java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExec
  utor.java:1164)
  java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExe
  cutor.java:634)
  com.ibm.ws.objectgrid.thread.XSThreadPool$Worker.run(XSThreadPoo
  l.java:309)
  
  Followed by connection lost between Catalog and containers.
  
  00010def LeaderManager I CWOBJ7218I: This catalog server based
  on an explicit attempt at communication has determined that
  C000WXSP\C000WXSPAS603\C000WXSCONTAINERP603 is lost.
  
  Containers need to be restarted to recover service.
  

Local fix

• Limit number of clients started concurrently.
  

Problem summary

• ****************************************************************
  * USERS AFFECTED:  Users of eXtreme Scale who use the Spring   *
  *                  Cache integration, near cache invalidation  *
  *                  or continuous query functions.              *
  ****************************************************************
  * PROBLEM DESCRIPTION: When applications do not                *
  *                      appropriately destroy their             *
  *                      ObjectGrid instances, they remain in    *
  *                      memory and can cause problems.          *
  ****************************************************************
  * RECOMMENDATION:  Use a single ObjectGrid instance per        *
  *                  application instead of continually using    *
  *                  ObjectGridManager.connect() or destroy      *
  *                  ObjectGrid instances when they are          *
  *                  no longer needed by your application.       *
  ****************************************************************
  When using Spring Cache integration, near cache invalidation
  or continuous query functions, client ObjectGrid instances
  would not garbage collect when no longer referenced by an
  application.  It is the application's responsibility to destroy
  an ObjectGrid instance or disconnect a ClientClusterContext
  when the ObjectGrid instance or connection is no longer
  needed. However, Spring Cache function does not have the
  ability to destroy ObjectGrid instances or disconnect.  To
  handle applications not re-using ObjectGrid or Spring Cache
  instances appropriately or not destroying instances when done
  with them, the Spring Cache integration, near cache
  invalidation and continuous query functions were updated to
  utilize WeakReferences to allow for instances to appropriate
  garbage collect and notify containers that the client
  ObjectGrid instances are destroyed.
  

Problem conclusion

• An interim fix is available for this APAR upon request.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  WS EXTREME SCAL

• Fixed component ID

  5724X6702

Applicable component levels