Configuring Liberty collective replica sets

You can configure collective replica sets. A replica set provides highly available management capabilities for a Liberty administrative domain.

Before you begin

The collectiveController-1.0 feature and its capabilities are available only in WebSphere® Application Server Network Deployment Liberty and WebSphere Application Server for z/OS® Liberty. The feature is not available in WebSphere Application Server Liberty, or WebSphere Application Server Liberty Core. If you have a WebSphere Application Server Network Deployment Liberty installation, you can use its collectiveController-1.0 feature to work with collective members from WebSphere Application Server Liberty, or WebSphere Application Server Liberty Core installations. A collective member can be from any WebSphere Application Server Liberty release.

About this task

A replica set is a set of collective controllers that are configured to work together. To create a replica set, you increase the number of collective controllers and configure them so that they can communicate with each other. Each new collective controller is called a replica because the added collective controllers have the same security configuration as the original controller. All information written to any one controller is automatically replicated to all other active controllers. When configured, all collective controllers in the replica set can perform the same operations as the original controller. Each replica contains all the repository updates that the other replicas within the set have processed. Therefore, there is no need for a member or client to connect with a particular collective controller each time that it interacts with the collective. Any of the collective controllers that are configured in the replica set can provide the same data.

A replica set requires an odd number of replicas and the minimum number of controllers is three. This configuration provides high availability and makes it possible to maintain a consensus with a majority. All replicas and controllers in a set are co-equal. There are no primary or secondary replicas or controllers in a replica set.

During the life of a replica set, it is sometimes necessary to add one or more replicas to an existing set. In this case, the existing replicas in the set do not require any updates to their configuration. Optionally, you can update their configurations in the server.xml file so that the files more accurately reflect the replicas that form the replica set, but this update does not affect their behaviors.

Note: It is not necessary to change the replicaSet value in the server.xml file of an existing replica in the set. No change to the configuration of an existing replica is required. If you want to update the replicaSet values in the configurations of existing replicas in the set so that the configuration values are consistent across all replicas in the set, you must set the isInitialReplicaSet value of the existing replicas to false. After you alter the replicaSet value, it describes a changed replica set rather than the initial replica set.

For detailed instructions about creating and configuring a collective controller, see Configuring a Liberty collective.

For an example of how to create a replica set that consists of three collective controllers on the same host, see Example: Create and activate a replica set .

Note: Whenever you refer to a replica, you must be consistent and always use the same host:port value. If a host name or IP address is used in one instance, it must be used in all instances.

To add a replica to an existing replica set, follow these steps:

Procedure

Ensure that the existing replica set is running and that most of the replicas are available.
Create a server to be the new collective controller.
```
wlp/bin/server create MyNewController
```
Replicate to transform the new server into a collective controller.
```
wlp/bin/collective replicate MyNewController 
   --host=host_of_running_controller  
   --port=https_port_of_running_controller 
   --user=userName_for_running_controller 
   --password=userPassword_for_running_controller 
   --keystorePassword=keystore_password_for_new_controller
   --hostName=myHost.myCompany.com
   --autoAcceptCertificates=true
   --createConfigFile=c:/wlp/usr/servers/myNewController/collective-replicate-include.xml
```
To reduce the number of options needed, use the --controller parameter instead of the --user, --password, --host, and --port parameters.
```
wlp/bin/collective replicate MyNewController 
--controller=userName_for_running_controller:userPassword_for_running_controller@host_of_running_controller:https_port_of_running_controller 
--keystorePassword=keystore_password_for_new_controller
--hostName=myHost.myCompany.com
--autoAcceptCertificates=true
--createConfigFile=c:/wlp/usr/servers/myNewController/collective-replicate-include.xml
```
If the --createConfigFile=output_file_path parameter is specified, the replicate command writes XML output to a file, rather than a console screen. The XML output contains a configuration element, including public and private SSH key paths. If the collective-wide SSH keys that are used in the initial controller are auto-generated, the keys are copied to the replica server at the default location. Otherwise, the keys must be manually copied over to the replica server and the paths must reflect the location where they were copied.
Next, include the outputted file in the collective configuration by adding an include statement to the server.xml file:
```
<include location=output_file_path />
```
If the --createConfigFile=out_file_path parameter, is not specified, the replicate command writes XML output to a console screen. In this case, you must manually copy the output into the server.xml file.
If the --hostName=myHost.myCompany.com parameter is specified by the collective replicate command, add host="*" into the httpEndpoint stanza of the new controller's server.xmlfile.
If the --createConfigFile=output_file_path parameter is specified, configure the new replica's included file as specified by the include clause in step 3. If this parameter is not specified, configure the file in the server.xmlfile, by using output of the replicate command. You can modify the replica configuration by using the following settings:
- Required settings:
  These values must be explicitly set. The XML printed by the replicate command contains this configuration and is sufficient for these settings.
  
  collectiveHostAuthInfo
  
  Collective-wide SSH configuration. This configuration element defines the method of remote authentication that is used throughout the collective and must be identical to that of the original controller. The new controller must define this element. If the original controller is using auto-generated SSH keys, those keys are automatically copied to this new controller by the replicate command. Otherwise, the SSH keys must be manually copied to the new system and the paths that are defined in collectiveHostAuthInfo must reflect this location
  
  hostAuthInfo
  
  Host authentication information that contains properties that a remote client needs to start the server. This setting is only required when the host of the new replica is not enabled for SSH. Typically, Linux® hosts are enabled for SSH and Windows hosts are not enabled for SSH. Thus, this setting might be needed only on Windows hosts. Set RPC credentials for the replica in one of the following of two ways:
  
  Set the rpcUser value to an operating system login user ID for the host on which the replica resides, and set the rpcUserPassword value to the operating system login password for the user ID. For example, if you log in to the replica computer with user test1 and password test1pwd, then change the hostAuthInfo element to the following configuration:
  <hostAuthInfo rpcUser="test1" rpcUserPassword="test1pwd" />
  
  If the replica host is registered with the collective controller, set the hostAuthInfo useHostCredentials parameter to true if you want the replica to inherit RPC credentials from its host.
  <hostAuthInfo useHostCredentials="true" />
  
  See Overriding Liberty server host information for information about hostAuthInfo settings.
  
  replicaSet
  
  Space-delimited list that contains the host:port for each of the replicaHosts and replicaPorts in the replica set, excluding the values for the collective controller now that is being added to the set.
  For example,
  original.host.com:10001 some.other.host.com:10003
  At least one of the values in this set must already be a replica of the existing replica set.
- Optional settings:
  These settings have default values, which can be altered.
  
  isInitialReplicaSet
  
  A boolean value that specifies whether a collective controller replica is joining an existing replica set or defining the initial collective controller in a new replica set. The default setting is false.
  
  replicaHost
  
  Host name for each individual replica
  
  replicaPort
  
  Port for each individual replica
  This port is a unique port that is used for communication between the replicas of the replica set.
  
  repositoryDir
  
  Directory location that is used to store repository data
Here is an example of what you might add to a new replica server.xml file:
```
<collectiveController replicaHost="localhost" 
   replicaPort="10012" 
   replicaSet="localhost:10010 localhost:10011" 
   isInitialReplicaSet="false"/> 
```
The XML printed by the replicate command requires updating the server's security configuration and specifying the collectiveRootKeys keystore password. The server's security configuration must be identical to the original collective controller's configuration. Also, the collectiveRootKeys keystore password must be the password that is used for the original collective controller's collectiveRootKeys keystore password. If the replica was created from the controller that is created in Configuring a Liberty collective, the new controller configuration must contain the appropriate information.
The new controller configuration must contain the following information:
```
<quickStartSecurity userName="adminUser" userPassword="adminPassword" />
  
  <keyStore id="collectiveRootKeys" password="yourPassword"
    location="${server.config.dir}/resources/collective/rootKeys.p12" />
```
If you are using the JKS keystore through 19.0.0.2, the new controller configuration must contain the following information:
```
<quickStartSecurity userName="adminUser" userPassword="adminPassword" />
  
  <keyStore id="collectiveRootKeys" password="yourPassword"
    location="${server.config.dir}/resources/collective/rootKeys.jks" />
```
Start the new replica by starting the new collective controller.
Confirm that the original collective controller successfully connected to the new replica. Look for message CWWKX6009I in the messages.log files of both the original collective controller and the replica.

Tip: For scripts that run the replicate and addReplica commands, add a 10-second wait after the replicate command runs to ensure the original collective controller and replica have time to connect before the addReplica command runs.

Start the addReplica operation on the collective utility to activate the new replica. The argument to the addReplica set must be the endpoint of the new replica (in the form replicaHost:replicaPort).

wlp/bin/collective addReplica localhost:10012 
  --host=host_of_running_controller
  --port=port_of_running_controller
  --user=user_for_running_controller
  --password=user_password
  --autoAcceptCertificates=true

To reduce the number of options needed, use the --controller option instead of the --user, --password, --host, and --port options.

wlp/bin/collective addReplica localhost:10012 --controller=user_for_running_controller:user_password@host_of_running_controller:port_of_running_controller --autoAcceptCertificates=true

Example: Creating and activating a replica set

This example describes how to create and activate a replica set. A replica set must have at least three replicas, preferably on different hosts, for high availability. This example has the replicas on the same host, which requires that you assign unique port numbers for the replicas. When replicas are on different hosts, the replicas can use the same port numbers.

Create a replica set.
1. If you do not have a collective controller, create one. See step 1 of Configuring a Liberty collective.
2. Ensure that the existing collective controller is running. For an existing controller named myController, run the following status command:
```
wlp/bin/server status myController
```
  If the collective controller is not running, then start it using the start or run command:
```
wlp/bin/server start myController
```
3. Create a server to be the new collective controller.
```
wlp/bin/server create myController2
```
4. Replicate the existing collective controller configuration into the new collective controller. The new collective controller is called a replica.
  Run a replicate command that uses the administrative security domain configuration of the existing collective controller and sets a new keystore password for the replica. Look at the server.xml file of the existing collective controller to find the values for the --host, --port, --user, and --password parameters. For the --keystorePassword parameter, set a value to use for the keystore, such as myController2. For information about these required parameters and about optional parameters, run the collective help replicate command at a command line.
```
wlp/bin/collective replicate myController2 --host=host_of_existing_controller --port=https_port_of_existing_controller --user=userName_for_existing_controller --password=userPassword_for_existing_controller --keystorePassword=keystore_password_for_new_controller--hostName=localhost --autoAcceptCertificates=true --createConfigFile=c:/wlp/usr/servers/myNewController/collective-replicate-include.xml
```
  To reduce the number of options needed, use the --controller option instead of the --user, --password, --host, and --port options.
```
wlp/bin/collective replicate myController2 --controller=userName_for_existing_controller:userPassword_for_existing_controller@host_of_existing_controller:https_port_of_existing_controller --keystorePassword=keystore_password_for_new_controller --hostName=localhost --autoAcceptCertificates=true --createConfigFile=c:/wlp/usr/servers/myNewController/collective-replicate-include.xml
```
  If you are prompted to accept the certificate chain, enter y (yes).
  
  If the --createConfigFile=output_file_path parameter is specified, the replicate command writes XML output to a file, rather than a console screen. In this case, include the output file in the collective configuration by adding an include statement to the server.xml file:
```
<include location=output_file_path />
```
  If the --createConfigFile=output_file_path parameter is not specified, the replicate command writes XML output to a console screen. In this case, you must manually add the output into the server.xml file of the new replica.
  
  If the --hostName=myHost.myCompany.com parameter is specified by using the collective replicate command, add the host="*" parameter into the httpEndpoint stanza in the new controller's server.xml file.
5. Edit the parameter values of the XML output from the replicate command that is generated by the collective replicate command. These values might be in the server.xml file or the include file that is specified by the --createConfigFile=output_file_path parameter. If the --createConfigFile=output_file_pathparameter is not specified, edit the parameter values of the XML output in server.xml file of the new replica.
  - In the server.xml file, ensure that the httpEndpoint element sets replica httpPort and httpsPort values that are unique port numbers on the host. For example, suppose the original controller that is named myController and the replica are both on the same localhost and myController has the following the httpEndpoint element:
```
<httpEndpoint id="defaultHttpEndpoint"
              host="*"
              httpPort="9080"
              httpsPort="9443" />
```
    Change the values for myController2 to:
```
<httpEndpoint id="defaultHttpEndpoint"
              host="*"
              httpPort="9085"
              httpsPort="9448" />
```
  - Make the following changes in the include file that is specified by the --createConfigFile=output_file_path parameter. If the --createConfigFile=output_file_path parameter is not specified, make the changes in the server.xml file.
    - Set RPC credentials for hostAuthInfo. You can set RPC credentials for the replica in either of two ways:
      - Set hostAuthInfo RPC user and password values. Set rpcUser to an operating system login user ID for the host on which the replica resides, and set rpcUserPassword to the operating system login password for the user ID. For example, if you log in to the replica computer with user test1 and password test1pwd, then change the hostAuthInfo element to the following:
        <hostAuthInfo rpcUser="test1" rpcUserPassword="test1pwd" />
      - If the replica host is registered with the collective controller, set hostAuthInfo useHostCredentials to true for the replica to inherit RPC credentials from its host.
        <hostAuthInfo useHostCredentials="true" />
      See Overriding Liberty server host information for information about hostAuthInfo settings.
    - Ensure that replicaPort sets a port number for the replica that is unique among the replica set and that replicaSet sets host:port values that identify the replica set. For example, if the original controller named myController and the replica are both on the same localhost, change the values for myController2 from 10010:
```
<collectiveController replicaPort="null"
                      replicaSet="localhost:null"
                      isInitialReplicaSet="false" />
```
      to 10011 for the replica port and 10010 for the replica set port:
```
<collectiveController replicaPort="10011"
                      replicaSet="localhost:10010"
                      isInitialReplicaSet="false" />
```
    - Ensure that the security configuration sets the same values as those used by the original controller. For example, both myController and the myController2 replica use:
```
<quickStartSecurity userName="adminUser" userPassword="adminPassword" />
```
    - Ensure that the collective root signers keystore element sets the same password as that used by the original controller. For example, copy the collectiveRootKeys keystore password from the myController server.xml file and paste it into the myController2 replica server.xml file.
      This example shows a generated password:
       <keyStore id="collectiveRootKeys" password="{xor}Lz4sLCgwLTs=" location="${server.config.dir}/resources/collective/rootKeys.p12"/>
      The following example shows a generated password if you are using the JKS keystore through 19.0.0.2:
```

<keyStore id="collectiveRootKeys" password="{xor}Lz4sLCgwLTs="
          location="${server.config.dir}/resources/collective/rootKeys.jks"/>
```
6. Start the new replica by starting the new collective controller.
```
wlp/bin/server start myController2
```
7. Verify that the original collective controller can communicate with the new replica.
  1. Open an editor on the original controller messages log, $WLP_USER_DIR/servers/myController/logs/messages.log.
  2. Look for the following message, which might have different IP addresses in your environment:
```
CWWKX6009I: The collective controller successfully connected to replica 127.0.0.1:10011. Current active replica set is [127.0.0.1:10010]. The configured replica set is [127.0.0.1:10010]. The connected standby replicas are [127.0.0.1:10011].
```
8. Verify that the new replica can communicate with the original collective controller.
  1. Open an editor on the replica messages log, $WLP_USER_DIR/servers/myController2/logs/messages.log.
  2. Look for the following message, which might have different IP addresses in your environment:
```
CWWKX6009I: The collective controller successfully connected to replica 127.0.0.1:10010. Current active replica set is []. The configured replica set is []. The connected standby replicas are [127.0.0.1:10011, 127.0.0.1:10010].
```
    Additionally, expect to see the following message as the controller is not active currently. You can ignore the message.
```
CWWKX6008E: The collective controller is unavailable, probably due to a loss of majority of the replica set, or a communications failure.  Current active replica set is [127.0.0.1:10010]. The configured replica  set is [127.0.0.1:10010]. The connected standby replicas are  [127.0.0.1:10011]
```
Activate the new replica.
Run an addReplica command that uses the administrative security domain configuration of the collective controller and specifies the endpoint of the replica that you want to activate (in the form replicaHost:replicaPort). Look at the server.xml file of the collective controller to find the values for the --host, --port, --user, and --password parameters. Look at the server.xml file of the replica to find the values for the replicaHost:replicaPortparameters. For information about these parameters, run the collective help addReplica command at a command line.
```
wlp/bin/collective addReplica replicaHost:replicaPort --host=host_of_existing_controller --port=port_of_existing_controller --user=user_for_existing_controller --password=user_password --autoAcceptCertificates=true
```
To reduce the number of options needed, use the --controller option instead of the --user, --password, --host, and --port options.
```
wlp/bin/collective addReplica replicaHost:replicaPort --controller=user_for_existing_controller:user_password@host_of_existing_controller:port_of_existing_controller --autoAcceptCertificates=true
```
For this example, which has the existing collective controller and the replica on the same host, localhost, run the following command:
```
wlp/bin/collective addReplica localhost:10011 --host=localhost --port=9443 --user=adminUser --password=adminPassword --autoAcceptCertificates=true
```
To reduce the number of options needed, use the --controller option instead of --user, --password, --host, and --portoptions.
```
wlp/bin/collective addReplica localhost:10011 --controller=adminUser:adminPassword@localhost:9443 --autoAcceptCertificates=true
```
If prompted to accept the certificate chain, enter y (yes).

Repeat steps 1 and 2 for any other replicas that you want to create. For example, add a third replica to the replica set. Name the new replica myController3 and specify replicaPort="10012". To ensure high availability, a replica set must have at least three replicas.

After the third replica is added to the replica set, you can verify that the original collective controller and new replicas are synchronized successfully.

Look for the following messages in the original controller messages log. The messages might have different IP addresses in your environment:

CWWKX6015I: A request to change the active collective controller replica set was received and is now processing. The current active replica set is {127.0.0.1:10010,127.0.0.1:10011}. The requested new active replica set is {127.0.0.1:10010,127.0.0.1:10011,127.0.0.1:10012}.

CWWKX6016I: The active collective controller replica set changed successfully. The current active replica set is {127.0.0.1:10010,127.0.0.1:10011,127.0.0.1:10012}. The previous active replica set was {127.0.0.1:10010,127.0.0.1:10011}.

CWWKX6011I: The collective controller is ready, and can accept requests. The leader is 127.0.0.1:10010. Current active replica set is [127.0.0.1:10012, 127.0.0.1:10011, 127.0.0.1:10010]. The configured replica set is [127.0.0.1:10010, 127.0.0.1:10011, 127.0.0.1:10012].

CWWKX6014I: This collective controller replica finished synchronizing the data with the other replicas.

Look for the following messages in the messages logs of the added replicas. The messages might have different IP addresses in your environment:

CWWKX6016I: The active collective controller replica set changed successfully. The current active replica set is {127.0.0.1:10010,127.0.0.1:10011,127.0.0.1:10012}. The previous active replica set was {127.0.0.1:10010,127.0.0.1:10011}.

CWWKX6011I: The collective controller is ready, and can accept requests. The leader is 127.0.0.1:10010. Current active replica set is [127.0.0.1:10012, 127.0.0.1:10011, 127.0.0.1:10010]. The configured replica set is [127.0.0.1:10010, 127.0.0.1:10011, 127.0.0.1:10012].

CWWKX8112I: The server's host information was successfully published to the collective repository.

CWWKX8114I: The server's paths were successfully published to the collective repository.

CWWKX8116I: The server STARTED state was successfully published to the collective repository.