10.0

Edit

Version 10.1.2.2

Release date

May 29, 2020

This patch introduces new Worker Management Service strategy, enhancements for worker run lifecycle, network load optimizations, and monitoring improvements.

The following components are patched:

  • Control Tower
  • Worker Management Service
  • Control Tower Worker

Equal worker distribution across BEP cluster

Default Worker Management Service (WMS) strategy of creating workers populates BEP Agents with workers sequentially. First, WMS creates workers on the Agent 1 to fill up its available hardware capacity (CPU and RAM), then it moves on to fill up with workers the Agent 2 and so on. Changing workload on the BEP cluster in terms of the tasks for Control Tower and Machine Learning workers, the typical utilization of available Agent resources is not balanced, i.e. some BEP Agents are heavily occupied by workers whereas other BEP Agents are underutilized.

With the new worker management strategy of WMS introduced in this patch, the utilization of BEP Agents is optimized for available resources of each agent. The strategy now calculates which Agents have more resources and are more suitable for adding new workers. As unused workers get killed and resources get released, the strategy reevaluates available resources and adjusts target BEP agent where new workers are to be launched.

Improved Error mode

When the start of CT or ML workers fails several times in a row, TDS terminates all active and failed workers in a group and the system goes into the Error mode. WMS or Task Dispatcher Service sends the following message to the Business Process event log:

Worker could not be started! Check Artifact|Launch configuration. Error info:

The patch implements the following fixes:

  • If Marathon contains active running workers, the Error mode triggers and active workers are not terminated, even though there might be some of workers in the failed state.
  • The Error mode triggers when all workers of a particular type fail to start in the clear cluster state. That means, no worker of this type is present.
  • Worker Management Service now kills only failed workers.
  • Worker Management Service now logs errors both for failed workers and workers in the Error mode. All these messages can be found in Kibana.

Decreased network load

The updated CT Worker features the following improvements:

  • To reduce the network traffic, CT Worker now downloads and saves Bot Config Bundles (BCB) to a temporary local directory. BCB is stored there as a temporary file and is deleted automatically, when the worker is killed.

    To setup this behavior, in worker.yml for Control Tower, edit the setting -Dwebharvest.machine.config.bundle.dir=${temp.dir}/bcb/.

  • The default resource allocations for worker CT are changed:

    MaxMetaspaceSize = 1024M

xmx = 2048

cpu = 0.4

Improved system’s stability

The following Zookeper settings for WMS were updated to provide stable operation:

SettingValueDescription
bep_wms_scaling_scale_after:'40s'The time interval for WMS to wait before scaling workers.
bep_wms_scaling_max_new_workers:'5'Max number of new workers that WMS creates in one iteration of a scaling strategy.
bep_wms_scaling_period_max_workers:'60s'The time interval to return the maximum count of waiting tasks. For example, the method returns maximum waiting tasks for a worker group for the last 60s. It's used for calculating the number of new workers to be launched
bep_wms_scaling_strategy:'supernova'A new strategy name.
bep_wms_worker_stop_after:'60m'.How long at least one worker in the pool stays alive after the last task. It stays alive only if there is no contention (cluster can fit at least one worker).
bep_wms_worker_min_working_time:'20s'Min time that worker must exist. WMS does not kill worker earlier in any case

Monitoring enhancements

To reduce load on a shared disk, the default timeout setting of the MetricBeat workfusion module for scanning metrics is increased from 60 to 300 seconds. See parameter period in file INSTALL_DIR/metricbeat/modules.d/workfusion.yml.

note

For the update information, see the Upgrade from 10.1.2 to 10.1.2.2 guide.

This patch supports a rollback to the previous v10.1.2 release.

← Version 10.1.4Version 10.1.2.1 →