Edit

Throughput dashboards

[BEP] [Tasks by worker] Input Task

The dashboard displays how the amount of tasks in task queues changes over time.

Clients (Control Tower and AutoML) produce tasks to task queues, and Task Dispatcher Service (TDS) consumes the tasks from the task queues and forwards them to a Worker queue. Forwarding is limited by the TDS throughput and Worker queue backpressure—the maximum amount of messages in the queue (~ 3 * Worker throughput).

The metrics are aggregated by GAVP—the number of task messages in all task queues related to a specific Worker pool.

[BEP] TDS throughput by worker type

The dashboard displays the TDS throughput in messages per second. It can be limited by the backpressure for the Worker queue, which is also implemented in TDS. TDS tries to maintain a small number of messages in the Worker queue (current workers throughput * bep.tds.broker.throttle-ahead).

The metrics are aggregated by GAVP.

[BEP] [Tasks by worker] Worker Tasks

The dashboard displays the count of a task in a Worker queue. TDS forwards tasks to the Worker queue, and Workers consume them.

The metrics are aggregated by GAVP.

[BEP] Workers throughput by worker type

The dashboard displays Worker throughput in messages per second.

The metrics are aggregated by GAVP.

[BEP] [Workers by Type] Worker Count

The dashboard displays the count of Workers.

The metrics are aggregated by GAVP + groupId.

[BEP] [Tasks By Client] Result Tasks

The dashboard displays the results produced by Workers and consumed by a client.

The metrics are aggregated by clients.

[BEP] Client throughput

The dashboard displays the throughput of a client app. It is the same as the consumption speed from the result queue.

The metrics are aggregated by clients.

Tips to identify bottlenecks

To find bottlenecks, take into account the following considerations:

• If Worker Tasks is almost empty (the count of messages in Worker Tasks is less than in Workers Throughput) and Input Tasks is not empty, the bottleneck is TDS. Try increasing bep.tds.broker.prefetch-count by 3 to 5 times.
• If Worker Tasks is almost empty (the count of messages in Worker Tasks is less than in Workers Throughput) and Input Tasks is also almost empty, the bottleneck is the client (Control Tower).
• If Worker Tasks is not empty (the count of messages in Worker Tasks is more than 2 x Workers Throughput), the Workers' processing is the bottleneck. Adding Agent nodes can increase the throughput.