PM98048: In an MMR environment a large number of ConcurrentModificationException exceptions may occur.

APAR status

• Closed as program error.

Error description

• A large number of ConcurrentModificationExceptions can occur
  when WXS tries to update replicas.
  
  The exact number of the exceptions thrown may be checked in the
  FFDC exception.log.  The large amount of work generated by this
  failure may drive up CPU and GC, causing other issues as well.
  
  The ConcurrentModificationException can be seen in the FFDC:
  
  key = java.util.ConcurrentModificationException
  com.ibm.ws.objectgrid.replication.PrimaryShardImpl.updateActiveR
  eplicas 277
  Exception = java.util.ConcurrentModificationException
  Source =
  com.ibm.ws.objectgrid.replication.PrimaryShardImpl.updateActiveR
  eplicas
  probeid = 277
  Stack Dump = java.util.ConcurrentModificationException
   at
  java.util.HashMap$AbstractMapIterator.checkConcurrentMod(HashMap
  .java:122)
   at
  java.util.HashMap$AbstractMapIterator.makeNext(HashMap.java:127)
   at java.util.HashMap$EntryIterator.next(HashMap.java:188)
   at java.util.HashMap$EntryIterator.next(HashMap.java:181)
   at
  com.ibm.ws.objectgrid.replication.PrimaryShardImpl.updateActiveR
  eplicas(PrimaryShardImpl.java:825)
   at
  com.ibm.ws.objectgrid.replication.PrimaryShardImpl.updateActiveR
  eplicas(PrimaryShardImpl.java:321)
   at
  com.ibm.ws.objectgrid.replication.ReplicatedPartition.processPri
  maryWork(ReplicatedPartition.java:577)
   at
  com.ibm.ws.objectgrid.replication.ReplicatedPartition.handleMigr
  ationEvent(ReplicatedPartition.java:518)
   at
  com.ibm.ws.objectgrid.ShardImpl.handleRoleTransition(ShardImpl.j
  ava:410)
   at
  com.ibm.ws.objectgrid.container.ObjectGridContainerImpl.doContai
  nerWork(ObjectGridContainerImpl.java:806)
   at
  com.ibm.ws.objectgrid.container.ObjectGridContainerImpl.access$1
  00(ObjectGridContainerImpl.java:144)
   at
  com.ibm.ws.objectgrid.container.ObjectGridContainerImpl$2.run(Ob
  jectGridContainerImpl.java:674)
   at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1551)
  

Local fix

Problem summary

• ****************************************************************
  * USERS AFFECTED:  WebSphere eXtreme Scale users who have a    *
  *                  multi-master replication (MMR)              *
  *                  configuration.                              *
  ****************************************************************
  * PROBLEM DESCRIPTION: When processing a remote MMR domain     *
  *                      and the local domain, a loop on the     *
  *                      ConcurrentModificationException         *
  *                      exception can occur.                    *
  ****************************************************************
  * RECOMMENDATION:                                              *
  ****************************************************************
  When WebSphere eXtreme Scale processes partition work from a
  remote domain and the local domain at the same time, the work
  might conflict. As a result, a ConcurrentModificationException
  exception occurs. After the exception occurs, the partition
  remains busy, looping on the error, and does not process
  more placement work. A large number of CWOBJ4815 and CWOBJ4816
  messages might result when the placement work cannot be
  completed.
  

Problem conclusion

• WebSphere eXtreme Scale now correctly processes the
  concurrent placement activities that cause the
  ConcurrentModificationException exception.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  WS EXTREME SCAL

• Fixed component ID

  5724X6702

Applicable component levels

• R850 PSY

     UP

• R860 PSY

     UP