Configuring request and retry timeout values

[Version 8.6.0.2 and later] You can supply tuning options to control how long the eXtreme Scale client code waits for requests to complete, and for how long it attempts to retry requests that are associated with accessing the data grid.

About this task

You can configure settings for the eXtreme Scale client that control how long the client attempts to create network connections, how long the client attempts to process a data grid request to a partition, and how long it attempts to retry that request to the partition, before it returns an exception to your application.

Factors for tuning request and retry timeout values in IBM eXtremeIO (XIO) and Object Request Broker (ORB)

For some tuning options, where you set the values depends on which transport you are using, either XIO or ORB. These transport-level tuning options have the initial impact on interactions with your client because they govern how long the transport attempts network socket connections and how long an individual remote procedure call (RPC) analogous to a data grid operation is given to complete.

When you tune these values, consider what your environment can tolerate under peak load conditions as well as steady state conditions. If you tune the intervals too far under the default values (30 seconds for request timeout, for example), your operations might fail prematurely. Consider the following factors:

Network latencies
Coupling of grid interactions with external resources like databases
Garbage collection pauses resulting from your combination of heap size, heap usage, and garbage collection tuning policies

ORB settings for tuning request and retry timeout values

The following timeout settings exist for the ORB:

com.ibm.CORBA.ConnectionTimeout: Specifies the amount of time that the ORB attempts to create a socket connection with the remote location before the attempts time out. The ORB caches these connections, and therefore, this operation is not done on every request.
com.ibm.CORBA.RequestTimeout: Specifies the amount of time that the ORB waits for an RPC to complete before timing out.
com.ibm.CORBA.FragmentTimeout: Reference the IBM ORB documentation for precise details. The product provides default settings for this value.
com.ibm.CORBA.LocateRequestTimeout: Reference the IBM ORB documentation for precise details. The product provides default settings for this value.
com.ibm.CORBA.SocketWriteTimeout: Specifies how many seconds a socket write waits before giving up.

As you tune the RequestTimeout and ConnectionTimeout settings, adjusting them based on the default recommendations can be appropriate. You can also set these settings with the same value, where you define these settings that are based on how long you want the request timeout to be.

XIO settings for tuning request and retry timeout values

With XIO, the following consolidated settings exist:

The xioTimeout setting determines how long the XIO transport attempts to establish a network socket connection.
There is no equivalent to the LocateRequest setting and the FragmentTimeout setting in the ORB.
The xioRequestTimeout value specifies how many seconds any request waits for a response before giving up. This property influences the amount of time a client takes to fail over if a network outage failure occurs. If you set this property too low, requests might time out inadvertently. Carefully consider the value of this property to prevent inadvertent timeouts.

Common settings for tuning request and retry timeout values

The next level of tuning is the requestRetryTimeout. With each transport type, after it throws a system exception because an RPC did not complete in time, the eXtreme Scale client can use the additional time that is defined by the requestRetryTimeout setting (for example, the request timeout is 10 seconds, and the retry request timeout is 20 seconds) to specify how long it takes to complete the following actions:

Asynchronously asks the catalog server for the latest routing table in case partitions are located elsewhere because of a failover.
Takes new routes and retries the request, or stops trying and throws an exception to your application.

The requestRetryTimeout property is set in milliseconds. Set the value greater than zero for the request to be retried on exceptions for which retry is available. Set the value to 0 to fail without retries on exceptions. To use the default behavior, remove the property or set the value to -1.

[Version 8.6.0.6 and later] XIO failture detection

The properties, xioRequestTimeout, xioTimeout, and requestRetryTimeout have an impact on the XIO failure detection system, in that the clients will be quicker to tell the catalog that a container might be failing, and therefore, trigger the catalog to attempt communication with the container. Where a failure exists, shard failure recovery is initiated for the container shards. Similarly, catalog calls to containers over XIO are governed by the xioRequestTimeout and xioTimeout properties.

Ways to set request retry timeout

You can configure the request retry timeout value on the client properties file or in a session. The session value overrides the client properties setting. If the value is set to greater than zero, the request is tried until either the timeout condition is met or a permanent failure occurs. A permanent failure might be a DuplicateKeyException exception. A value of zero indicates the fail-fast mode setting and the data grid does not attempt to try the transaction again after any type of transaction.

Transaction timeout and request retry timeout

During run time, the transaction timeout value is used with the request retry timeout value, ensuring that the request retry timeout does not exceed the transaction timeout.

Two types of transactions exist: Autocommit transactions, and transactions that use explicit begin and commit methods. The valid exceptions for retry differ between these two types of transactions:

For transactions that are called within a session, transactions are tried again for ORB CORBA SystemException (TransportException for XIO) and eXtreme Scale client TargetNotAvailable exceptions.
Autocommit transactions are tried again for CORBA SystemException and eXtreme Scale client availability exceptions. These exceptions include the ReplicationVotedToRollbackTransactionException, TargetNotAvailable, and AvailabilityException exceptions.

Application or other permanent failures return immediately and the client does not try the transaction again. These permanent failures include the DuplicateKeyException and KeyNotFoundException exceptions. Use the fail-fast setting to return all exceptions without trying transactions again after any exceptions.

Exceptions where the client tries the transaction again:

ReplicationVotedToRollbackTransactionException (only on autocommit)
TargetNotAvailable
org.omg.CORBA.SystemException (TransportException is the XIO equivalent of this ORB system exception.)
AvailabilityException (only on autocommit)
LockTimeoutException (only on autocommit)
UnavailableServiceException (only on autocommit)

Permanent exceptions, where the transaction is not tried again:

DuplicateKeyException
KeyNotFoundException
LoaderException
TransactionAffinityException
LockDeadlockException
OptimisticCollisionException

Procedure

Set the request retry timeout value in a client property file.
To set the requestRetryTimeout value on a client, add or modify the requestRetryTimeout property in the Client properties file. The client properties is the objectGridClient.properties file by default. The requestRetryTimeout property is set in milliseconds. Set the value greater than zero for the request to be retried on exceptions for which retry is available. Set the value to 0 to fail without retries on exceptions. To use the default behavior, remove the property or set the value to -1. An example of the value in the objectGridClient.properties file follows:
```
requestRetryTimeout = 30000
```
The requestRetryTimeout value is specified in milliseconds. In the example, if the value is used on an ObjectGrid instance, the requestRetryTimeout value is 30 seconds.
Set the request retry timeout value programmatically.
To set the client properties programmatically, first create a client properties file in an appropriate <location> for your application. In the following example, the client properties file refers to the objectGridClient.properties snippet in the previous section. After you connect to ObjectGridManager instance, set the client properties as described. Then, when you have an ObjectGrid instance, the instance has the client properties that you defined in the file. If you change the client properties file, you must explicitly get a new ObjectGrid instance each time.
```
ObjectGridManager manager = ObjectGridManagerFactory.getObjectGridManager();
String objectGridName = "testObjectGrid";
URL clientXML = null;
ClientClusterContext ccc = manager.connect("localhost:2809", null, clientXML);
File file = new File("<location>/objectGridClient.properties");
URL url = file.toURI().toURL();
ccc.setClientProperties(objectGridName, url);
ObjectGrid objectGrid = ogManager.getObjectGrid(ccc, objectGridName);
```
Set the override file during a session commit.
To set the request retry timeout on a session or to override the requestRetryTimeout client property, call the setRequestRetryTimeout(long) method on the Session interface.
```
Session sessionA = objectGrid.getSession();
sessionA.setRequestRetryTimeout(30000);
ObjectMap mapA = sessionA.getMap("payroll");
String key = "key:" + j;
mapA.insert(key, "valueA");
```
This session now uses a requestRetryTimeout value of 30000 ms or 30 seconds, regardless of the value that is set in the client properties file. For more information about the session interface, see Using Sessions to access data in the grid.

Example

Consider the following example, where the client can handle network latency, garbage collection, general contention on the server as a result of setting short timeout values. The requestRetryTimeout property is 10 seconds, and the xioTimeout property matches the ORB ConnectionTimeout value, which is 5 seconds.

Table 1. Data grid configurations for ORB and eXtremeIO transport types
Grid Type	ORB	XIO
A Java™ or .NET client application that accesses an eXtreme Scale API directly	Modify the orb.properties file for your client application. Set the following values: com.ibm.CORBA.RequestTimeout=5 com.ibm.CORBA.ConnectTimeout=5 com.ibm.CORBA.FragmentTimeout=5 com.ibm.CORBA.LocateRequestTimeout=5 com.ibm.CORBA.SocketWriteTimeout=5 Note: With WebSphere Application Server, you control the ORB settings through the deployment manager, and not through an orb.properties file. Modify the objectGridClient.properties file for the client application with requestRetryTimeout=10000.	Modify the objectGridClient.properties file for your client application with the following values: xioRequestTimeout=5000. This value is in milliseconds and is equivalent to com.ibm.CORBA.RequestTimeout. xioTimeout=5. This value is in seconds and is equivalent to com.ibm.CORBA.ConnectTimeout. requestRetryTimeout=10000. This value is in milliseconds and is also used for the ORB transport. ORB FragmentTimeout and LocateRequestTimeout have no XIO equivalent values.
HTTP session	Same ORB configuration as a Java or .NET client application that accesses an eXtreme Scale API directly	Same XIO configuration as a Java or .NET client application that accesses an eXtreme Scale API directly
Dynamic cache	Same ORB configuration as a Java or .NET client application that accesses an eXtreme Scale API directly. For dynamic cache instances, you can set the following additional property on the cache instance: `com.ibm.websphere.xs.dynacache. request_retry_timeout_override=10000` You can use the requestRetryTimeout setting with the client properties in the class path instead of this cache instance property, if you want it to be the same for every dynamic cache instance.	Same XIO configuration as a Java or .NET client application that accesses an eXtreme Scale API directly. For dynamic cache instances, you can set the following additional property on the cache instance: `com.ibm.websphere.xs.dynacache. request_retry_timeout_override=10000` You can use the requestRetryTimeout setting with the client properties in the class path instead of this cache instance property, if you want it to be the same for every dynamic cache instance.