Issues related to slow execution on Workers' side
You can observe delays in task execution due to tasks waiting long for a free Worker in a cluster or delays in Worker starts due to cluster resources utilized by other task types, thus being insufficient.
To investigate, act as follows:
Analyze Worker processing speed
To confirm that slow Worker processing is an issue, go to Kibana's Welcome dashboard and find the Task time-trace by BP Identifier dashboard.
Select a date range and check whether the two values below look normal or are noticeably high:
Task: waiting for worker
Task: processing by worker
You can also take a look at the [BEP] Tasks trace by BP steps dashboard and conduct an analysis as described in the Investigate Bot Task execution topic.
When the issue is identified as slow Worker processing, proceed with the next steps.
Check Worker availability
Identify if the issue is related to no available Workers to handle an incoming task, or the resource utilization in the cluster does not allow sending the task to a Worker for processing.
When Business Process (BP) performance is poor due to slow Worker execution, try identifying possible bottlenecks slowing down the Workers processing the code.
To check the cluster utilization, go to Kibana and check the Cluster utilization dashboard.
In the dashboard, you can see the resourse consumption by cluster components, such as AutoML, CT Workers, and so on:
Many scenarios are possible, but the most frequent issues are as follows:
- Network traffic, for instance, due to security bottlenecks, sending long logs or large-size OCR results over the network
- Verbose logging, for example, due to resources consumed by frequent exception stack trace logging
- Database slowdowns, for instance, due to a background data purging
Output
If you confirm the issue, create a ticket and provide the Support team with the collected information, for instance, dashboard screenshots. Otherwise, continue the investigation.
View also: