PI14077: Replicas time out during creation (CWOBJ1579W message) and are not added to the route table in larger environments.

APAR status

• Closed as program error.

Error description

• In larger environments with significant amounts of placement
  and more sparse CPU cycles, the creation of a replica by the
  primary could time out. The replica can still be created.
  However, it is not reported to the route table.
  
  This can be seen in the CWOBJ1579W message.  This indicates
  that the asynchronous replica took a long time to start, and
  the primary shard skips adding it to the route table; for
  example:
  
  PrimaryShardI W   CWOBJ1579W: Primary Grid:mapSet:7895 attempted
  to place a (asynchronous replica) on <container_name>. The
  remote container timed out before completing the task.
  
  Check <container_name> container server for confirmation of
  asynchronous replica creation.
  

Local fix

Problem summary

• ****************************************************************
  * USERS AFFECTED:  WebSphere eXtreme Scale users running with  *
  *                  asynchronous replicas that take several     *
  *                  seconds to initialize.                      *
  ****************************************************************
  * PROBLEM DESCRIPTION: Asynchronous replicas that take longer  *
  *                      than the ORB requestTimeout or XIO      *
  *                      xioTimeout do not get added             *
  *                      to the route table.                     *
  ****************************************************************
  * RECOMMENDATION:                                              *
  ****************************************************************
  The primary shard waits for an asynchronous replica to
  complete initialization, but the request times out. The
  replica is online, but is missing from the route table,
  which can be seen in the output from the xscmd command
  route table. Creating several template maps (for example, 100
  more) can cause an asynchronous replica to take a long time to
  initialize.
  The following message occurs on the primary shard container
  server when an async replica times out:
  PrimaryShardI W   CWOBJ1579W: Primary Grid:mapSet:0 attempted
  to place a (asynchronous replica) on server1_C-2. The remote
  container timed out before completing the task. Check
  server1_C-2 container server for confirmation of asynchronous
  replica creation.
  

Problem conclusion

• The xscmd triggerPlacement command resolves some route table
  problems. If triggerPlacement does not resolve, restarting the
  container with the replica shard can resolve the problem, if
  the long initialization was temporary or caused by system
  contention. Review the xscmd showPlacement output for the
  location of the asynchronous replica. Then, apply the
  interim fix.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  WS EXTREME SCAL

• Fixed component ID

  5724X6702

Applicable component levels

• R860 PSY

     UP