Skip to main content
Version: 10.3.1

Perform cluster autoscaling

To support cases with a non-constant load of available cluster resources and allow reducing TCO, you require elastic cluster scaling. Virtual machines (VM) in the cluster are added on demand and reduced when there is no load and they are no longer needed.

Work.AI works with a predefined amount of available cluster resources where each VM has some resource capacity (CPU, memory, disc space, and so on). These resources are used to deploy worker applications processing tasks. The amount of each type of these worker apps is calculated according to the current demand or the number of tasks for each type that should be processed considering the average throughput of a single BEP Worker.

Scaling is needed when:

  • Resources are underutilized, and only a few tasks are performed. In this case, the number of active VMs should be reduced.

  • Resources are overutilized, and the task load is heavy. Then, additional VMs should be added to process the load efficiently and increase throughput.

Two separate mechanisms are used to address these needs:

  • The BEP platform or Worker Management Service (WMS) tracks the task load and distribution of worker applications per cluster and makes decisions on when to scale up or down.

  • A special scaling service measures resource utilization for each VM and starts a new VM or shuts down the existing one.

Physical resource orchestration​

The WMS provides information on VM states to the scaling service. The WMS cluster resource controller has the following properties:

  • lookup_window: a defined amount of time to search through the latest scaling iteration status.

  • max_node_load_percent: a percent of total resource capacity (memory and CPU) occupied by BEP Workers to consider the load.

  • cool_down_interval: an interval to wait for the VM shutdown after releasing resources; must have a greater value than the physical resource orchestrator.

  • minimal_active_node_count: a number of VMs that should always be active; must have the same value as the physical resource orchestrator and equal or greater than 1.

The WMS provides the current autoscaling state via REST API (the worker-management-service/api/metrics/cluster/state endpoint). The response is a JSON object that contains the current autoscaling status and the list of VM hostnames for downscaling (if any). There are the following states:

  • KEEP: no actions are needed.

  • UPSCALE: a VM needs to be added.

  • DOWNSCALE: a VM needs to be removed.

For more details on each state and related actions, see Scaling | Service workflow.

Scaling​

The Work.AI scaling service is responsible for tracking data from the WMS and communicating with the cloud hypervisor to add or remove VMs in the cluster.

Authorization in cloud​

The scaling service is authenticated and gets authorized access to resources through a role or a service account attached to the VM where the service is running. It is the preferred method for finding credentials in production.

Service workflow​

VMs are added and removed based on a state received from the WMS. The workflow is as follows:

  1. The service polls the WMS every minute. You can change the polling time using the poll_interval value.

  2. The service gets a state from the poll results and performs some actions depending on this state:

    • KEEP means that the number of VMs corresponds to the current cluster load. No actions are needed.
    • DOWNSCALE shows that the cluster is underloaded. The message from the WMS contains the IP address of a VM that should be downscaled. Based on this, the scaling service finds the needed VM and shuts it down.
    • UPSCALE means the cluster is overloaded. The scaling service starts a procedure for creating a new VM based on the existing VM template. All the necessary startup scripts are run automatically, allowing the new VM to find and join the cluster. The WMS sees an additional agent and distributes the load accordingly.
    note

    You can set up the maximum number of VMs using the max_agents_count value so as not to upscale continuously. If the number of VMs does not exceed the specified amount, the upscale procedure starts.

  3. The procedure is repeated according to the poll interval time.