Defining scaling policies to manage workload

Scaling policies are used to start and stop dynamic cluster members based on configurable metrics. Scaling Policy can be defined for all or specific clusters. If no policy is defined, a built-in policy will be used.

About this task

The scaling controller uses scaling policies to make decisions about managing workload. By default, a built-in scaling policy is embedded in the scaling controller. Optionally, you may override the built-in scaling policy by defining either or both top-level elements, default scaling policy and scaling policy. The built-in scaling policy is as follows:

A minimum of two cluster members, if available, are kept active. The minimum number might not be met if some or all of the members are exceeding the metric thresholds.
An additional cluster member is started when the average CPU, heap, or memory use of all active members exceeds 90%.
A cluster member is stopped when the average CPU and heap use drops less than 30%.

Scaling policy may be defined for all clusters using the default scaling policy. The default scaling policy inherits built-in scaling policy metrics, including min and max instance values. Any changes made to the default scaling policy, specified by the user, will override the built-in value. Values that are not specified in your default scaling policy will be inherited from the built-in scaling policy.

Unlike default scaling policy, scaling policy metrics are not inherited from either the built-in or default scaling policy. However, min and max instance values are initialized to the built-in policy values. Because scaling policy metrics are considered optional values, only the metrics specified in the policy are included in scaling decisions. Metrics not included in the scaling policy are not analyzed when making scaling decisions.

There are two levels of scaling policies. You can override the built-in scaling policy by defining either or both top-level elements. Here are the two levels that can be defined:

Default scaling policy
You can define a single default scaling policy to manage all clusters that do not need a more specific scaling policy defined.
The following example shows how to define a default scaling policy that sets the minimum number of active cluster members to 3.
```
<scalingDefinitions>
 <defaultScalingPolicy min="3"/>
</scalingDefinitions>
```
Scaling policy
You can define a scaling policy to manage one or more clusters with targeted criteria that is specified in the scaling policy. Because scaling policy definitions do not inherit metric thresholds, only metric thresholds that are specified will be monitored. All other thresholds will be ignored as it relates to scaling decisions. Built-in scaling policy values for min and max instances will still be respected if they are not specified.
The following example shows how to define a scaling policy that changes the CPU use thresholds for starting and stopping servers in a cluster named cluster1.
```
<scalingDefinitions>
  <scalingPolicy id="cluster1Policy">
  <bind clusters="cluster1"/>
  <metric name="CPU" min="10" max="70"/>
 </scalingPolicy>
</scalingDefinitions>
```
The value of the clusters attribute of the bind element is a comma-separated list of cluster names. An asterisk can be used at the end of the name only as a wildcard to match zero-or-more characters. For example:
```
<bind clusters="west,south*"/>
```
In this example, the scaling policy applies to a cluster named west and all clusters whose name starts with south. If a cluster name matches multiple policies, the scaling controller uses the following rules to choose a policy:
- An exact match is preferred to a wildcard match.
- If there are multiple wildcard matches, the policy with the longest prefix is used.

The policy will scale based on default min and max values. In the previous example, the clusters will only scale based on the CPU values specified, where min=10 and max=70.

Metric-based scaling decisions are made on a cluster level. Each cluster member monitors its own metrics. The CPU and Memory metric value for a server is actually the greater of the server JVM process or the server's host. The Heap metric value is only taken from the server JVM process. If a measurable change in a monitored metric is detected, that metric is sent to the controller for analysis. All cluster member metrics are accumulated and a cluster average is calculated for each metric. The calculated value for each metric is then compared to the defined upper or lower threshold to determine if a scaling decision will be triggered.

Scale out decisions are made on an individual metric basis. All monitored metrics are analyzed for the cluster and if any one of the metrics exceeds the policy max threshold, a scale out event is triggered. Scale in decisions are made based on all monitored metrics. The cluster average for each monitored metric is analyzed. If all metrics are less than their policy min threshold, a scale in event is triggered.

After a scaling decision is made, the scaling controller selects a scaling target. The scaling target is the host on which the server stop (for scale in) or server start (for scale out) action is run. When determining a target for a scale out action, the host level metrics are taken into consideration. If any of the host level metrics exceed that metric's policy max threshold, the scaling controller avoids that host and chooses another host for the scale out action.

The scaling policy can influence the selection of the scaling target. If the policy specifies scalingPreference="horizontal", the scaling controller starts a server on the eligible host with the fewest active servers and stops a server on the host with the most active servers. If the policy specifies scalingPreference="vertical", the scaling controller starts a server on the eligible host with the most active servers and stops a server on the host with the fewest active servers. The scaling controller attempts to place servers in the same cluster on different hosts when possible, regardless of which scalingPreference is chosen.

For more information on scalingPolicy metrics see, Scaling Controller.

You can use the hostGroup element to limit which hosts can be selected to provision a new server. The tags attribute specifies a list of administrative metadata tags. When the scaling controller needs to provision a server, it can only select a host that has at least one matching tag. For example, the following policy says that servers can be provisioned on a host that has tag1 or a host that has tag2 associated with it.


<scalingDefinitions>        
  <scalingPolicy id="provisionPolicy">           
   <bind clusters="defaultStackGroup.deployMember"/>           
   <hostGroup tags="tag1 tag2"/>        
 </scalingPolicy>    
</scalingDefinitions>

Note: If you do not specify the hostGroup element, the scaling controller can provision a server on any host that is registered with the collective.

For more information on setting administrative metadata tags see, Setting administrative metadata for Liberty resources.

Procedure

In the server.xml file of the scaling controller, add the default scaling definitions.


<scalingDefinitions>
 <defaultScalingPolicy enabled="true" min="1" max="2"/>
</scalingDefinitions>

When a scaling policy is modified, the server.xml in the controller is also modified. You can check the messages.log file for the following messages to verify that the controller was successfully updated.

CWWKG0016I: Starting server configuration update.
CWWKG0028A: Processing included configuration resource: /opt/IBM/Liberty/wlp/usr/
CWWKG0017I: The server configuration was successfully updated in 0.052 seconds

You will see the following error messages in the messages.log file if the scaling policy is missing necessary metric thresholds. Follow the user action to correct the error.

CWWKV0126W: The configuration of scaling policy {0} does not contain a min or max threshold.
CWWKV0127W: The configuration of scaling policy {0} does not contain a max threshold.
CWWKV0128W: The configuration of scaling policy {0} does not contain a min threshold.

Results

Your scaling policies are now defined.