Defining scaling policies to manage workload
Scaling policies are used to start and stop dynamic cluster members based on configurable metrics. Scaling Policy can be defined for all or specific clusters. If no policy is defined, a built-in policy will be used.
About this task
- A minimum of two cluster members, if available, are kept active. The minimum number might not be met if some or all of the members are exceeding the metric thresholds.
- An additional cluster member is started when the average CPU, heap, or memory use of all active members exceeds 90%.
- A cluster member is stopped when the average CPU and heap use drops less than 30%.
Scaling policy may be defined for all clusters using the default scaling policy. The default
scaling policy inherits built-in scaling policy metrics, including min
and
max
instance values. Any changes made to the default scaling policy, specified by
the user, will override the built-in value. Values that are not specified in your default scaling
policy will be inherited from the built-in scaling policy.
Unlike default scaling policy, scaling policy metrics are not inherited from either the built-in
or default scaling policy. However, min
and max
instance values
are initialized to the built-in policy values. Because scaling policy metrics are considered
optional values, only the metrics specified in the policy are included in scaling decisions. Metrics
not included in the scaling policy are not analyzed when making scaling decisions.
- Default scaling policy
You can define a single default scaling policy to manage all clusters that do not need a more specific scaling policy defined.
The following example shows how to define a default scaling policy that sets the minimum number of active cluster members to 3.<scalingDefinitions> <defaultScalingPolicy min="3"/> </scalingDefinitions>
- Scaling policy
You can define a scaling policy to manage one or more clusters with targeted criteria that is specified in the scaling policy. Because scaling policy definitions do not inherit metric thresholds, only metric thresholds that are specified will be monitored. All other thresholds will be ignored as it relates to scaling decisions. Built-in scaling policy values for
min
andmax
instances will still be respected if they are not specified.The following example shows how to define a scaling policy that changes the CPU use thresholds for starting and stopping servers in a cluster namedcluster1
.<scalingDefinitions> <scalingPolicy id="cluster1Policy"> <bind clusters="cluster1"/> <metric name="CPU" min="10" max="70"/> </scalingPolicy> </scalingDefinitions>
The value of theclusters
attribute of thebind
element is a comma-separated list of cluster names. An asterisk can be used at the end of the name only as a wildcard to match zero-or-more characters. For example:
In this example, the scaling policy applies to a cluster named west and all clusters whose name starts with south. If a cluster name matches multiple policies, the scaling controller uses the following rules to choose a policy:<bind clusters="west,south*"/>
- An exact match is preferred to a wildcard match.
- If there are multiple wildcard matches, the policy with the longest prefix is used.
The policy will scale based on default min
and max
values. In
the previous example, the clusters will only scale based on the CPU values specified, where
min=10
and max=70
.
Metric-based scaling decisions are made on a cluster level. Each cluster member monitors its own metrics. The CPU and Memory metric value for a server is actually the greater of the server JVM process or the server's host. The Heap metric value is only taken from the server JVM process. If a measurable change in a monitored metric is detected, that metric is sent to the controller for analysis. All cluster member metrics are accumulated and a cluster average is calculated for each metric. The calculated value for each metric is then compared to the defined upper or lower threshold to determine if a scaling decision will be triggered.
Scale out decisions are made on an individual metric basis. All monitored metrics
are analyzed for the cluster and if any one of the metrics exceeds the policy max
threshold, a scale out event is triggered. Scale in decisions are made based on all
monitored metrics. The cluster average for each monitored metric is analyzed. If all metrics are
less than their policy min
threshold, a scale in event is triggered.
After a scaling decision is made, the scaling controller selects a scaling target. The scaling
target is the host on which the server stop (for scale in) or server start (for scale out) action is
run. When determining a target for a scale out action, the host level metrics are taken into
consideration. If any of the host level metrics exceed that metric's policy max
threshold, the scaling controller avoids that host and chooses another host for the scale out
action.
The scaling policy can influence the selection of the scaling target. If the policy specifies
scalingPreference="horizontal"
, the scaling controller starts a server on the
eligible host with the fewest active servers and stops a server on the host with the most active
servers. If the policy specifies scalingPreference="vertical"
, the scaling
controller starts a server on the eligible host with the most active servers and stops a server on
the host with the fewest active servers. The scaling controller attempts to place servers in the
same cluster on different hosts when possible, regardless of which
scalingPreference
is chosen.
For more information on scalingPolicy metrics see, Scaling Controller.
<scalingDefinitions>
<scalingPolicy id="provisionPolicy">
<bind clusters="defaultStackGroup.deployMember"/>
<hostGroup tags="tag1 tag2"/>
</scalingPolicy>
</scalingDefinitions>
For more information on setting administrative metadata tags see, Setting administrative metadata for Liberty resources.
Procedure
Results
Your scaling policies are now defined.