Distribute tasks in BEP

The figure below illustrates the BEP Worker task distribution flow within BEP:

Upon submission, each task is added to an input task queue. The Task Broker processes the tasks inside the Task Dispatcher Service and routes them from input queues to a Worker queue. The Task Broker listens to input queues from 1 to N, but routes tasks to a single Worker queue.

The Task Dispatcher Service runs multiple Task Brokers—one per Worker group. Workers in a particular group process tasks of a particular type (for details, see Worker types and default configurations). Although Worker processes are running on different nodes, they listen to the same Worker queue for tasks.

By separating the input and Worker queues, TDS can guarantee fair-share by round-robin selection and delivery of tasks to a proper Worker queue. To illustrate the benefits, let's assume you start one Business Proces (BP) with a million records, and after some time, you launch a second BP with 100 records. If you published tasks directly to a Worker queue, the first BP would fill up the Worker queue with one million tasks from the first step. The tasks from the second BP are to would be processed only after all the first-step records.

The task processing stages are as shown in the figure below: