PI05479: WHEN AN EXTREME SCALE SERVER ENCOUNTERS AN EXCEPTION THE REPLICA DOES NOT RECOVER.

APAR status

• Closed as program error.

Error description

• When an eXtreme Scale server encounters
  a DuplicateKeyException, KeyNotFoundException, the replica
  catalog server does not automatically recover.
  

Local fix

Problem summary

• ****************************************************************
  * USERS AFFECTED:  WebSphere eXtreme Scale users running with  *
  *                  two or more catalog servers.                *
  ****************************************************************
  * PROBLEM DESCRIPTION: If a WebSphere eXtreme Scale replica    *
  *                      catalog server encounters an error,     *
  *                      the replica catalog server does not     *
  *                      automatically recover. The replica      *
  *                      catalog server can have an inconsistent *
  *                      state compared to the primary catalog   *
  *                      server.                                 *
  ****************************************************************
  * RECOMMENDATION:                                              *
  ****************************************************************
  If a WebSphere eXtreme Scale replica catalog server encounters
  a DuplicateKeyException, KeyNotFoundException or other
  exception, the replica catalog server does not request a new
  and complete copy of data from the catalog server. When a
  replica catalog server encounters a transactional error, the
  replica catalog server must remove its bad state and request
  the correct state from the primary catalog server. A CWOBJ1524I
  message is in the JVM or SystemOut.log file of the replica
  catalog
  server when the state is removed and corrected.
  When a transaction error occurs on the replica catalog
  server, a CWOBJ1528 message displays in the JVM or SystemOut.log
  file.
  Example:
  [11/5/13 10:40:56:016 CST] 00000010 CommonReplica E
  CWOBJ1518E: An exception occurred when attempting to commit
  replica transaction (292369C7-4142-427F-E000-F0DEF1C45B9A) for
  primary transaction (<null>) on Replica (Replication Group
  Member {1} (Server[testCatalogReplicaDupKeyCat1]
  ObjectGrid[BalanceGrid] Mapset[ENTITY_MAPSET]
  ReplicationGroup[IBM_CATALOG_SERVER_REPLICATION_GROUP]
  Partition[0])):
  com.ibm.websphere.objectgrid.TransactionException: rolling
  back transaction, see caused by exception
  at
  com.ibm.ws.objectgrid.SessionImpl.rollbackPMapChanges(SessionImp
  l.java:2522)
  at
  com.ibm.ws.objectgrid.SessionImpl.commit(SessionImpl.java:2142)
  at
  com.ibm.ws.objectgrid.replication.CommonReplicationGroupMemberSe
  rvice.processRGMLogSequences(CommonReplicationGroupMemberService
  .java:1732)
  at
  com.ibm.ws.objectgrid.replication.CommonReplicationGroupMemberSe
  rvice.dispatchMessage(CommonReplicationGroupMemberService.java:7
  14)
  at
  com.ibm.ws.objectgrid.replication.CommonReplicationGroupMemberSe
  rvice.run(CommonReplicationGroupMemberService.java:858)
  at java.lang.Thread.run(Thread.java:736)
  Caused by: com.ibm.websphere.objectgrid.DuplicateKeyException:
  ObjectGrid: BalanceGrid, Map: EventQueue, Key: dupKey
  [dupKeysync]
  at
  com.ibm.ws.objectgrid.map.BaseMap.applyPMap(BaseMap.java:2245)
  at
  com.ibm.ws.objectgrid.SessionImpl.commit(SessionImpl.java:1753)
  ... 4 more
  

Problem conclusion

• Restart the replica catalog server with the CWOBJ1518E message
  if no CWOBJ1524I message follows (indicating automatic recovery)
  Apply the fix.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  WS EXTREME SCAL

• Fixed component ID

  5724X6702

Applicable component levels

• R850 PSY

     UP

• R860 PSY

     UP