Edit

Reduce RabbitMQ memory pressure

Problem

Control Tower creates a task queue for each Business Process (BP) step. When you have a large amount of active BPs with multiple steps, a situation may arise where RabbitMQ contains thousands of task queues.

The TDS task broker mechanism subscribes to all task queues to implement the fair-share task routing to the Worker queue. Each queue listener is a consumer with an open channel. In the case of a large number of open channels, this situation leads to RabbitMQ memory pressure. The memory pressure prevents RabbitMQ from receiving new tasks if the configured memory threshold gets breached.

Solutions

Adjust TDS parameters

The following adjustments were made to the TDS parameters to control the buildup of memory pressure:

1. Reduced the prefetch count of default messages by decreasing the bep.tds.broker.prefetch-count ZooKeeper parameter's value in TDS from 50 to 5. The default message prefetch count in TDS impacts memory consumption in RabbitMQ and TDS.
2. Increased the open channel limit from 2047 to 4096 and extended the memory threshold in product versions 10.1.6.2 and 10.2.2. When the memory threshold gets breached, TDS stops routing messages and produces log exceptions.
3. Decreased the GC interval to 24 hours.

Optimize BEP client

From product versions 10.1.6.2 and 10.2.2, WorkFusion has enabled creating lazy queues in CT by default. The lazy queues move their content to the disk as early as possible and load it in the RAM only when requested by consumers. This approach reduces RabbitMQ memory pressure and improves stability by increasing the maximum allowed number of queues, channels, and queue messages with the same memory requirements. It also takes less time to recover after breaching the memory threshold. To know more about this mechanism, visit https://www.rabbitmq.com/lazy-queues.html.