Plan infrastructure capacity
A common concern of an enterprise is whether it has enough allocated resources to handle an increase in users or the number of transactions. The capacity is defined as the maximum load that a service, system, or device can handle.
Infrastructure capacity planning is the process of determining the hardware and software resources required to meet the processing demand for the current moment and in the future.
Capacity Planning is a part of a wider Capacity Management domain. Capacity Management is concerned with ensuring the availability of a cost-effective capacity that meets or exceeds the business needs as established in Service Level Agreements at any given time.
Standard build
The System Requirements section summarizes the recommended standard specification for the environment to run WorkFusion Intelligent Automation Cloud Enterprise.
Several factors can impact the performance and scalability of the software. Important system variables include the following:
- Business Process design
- Server configuration
- Infrastructure tuning
- Data environment
- Compute capacity
- Networking
The numbers below were established in a closed network lab based on automated load tests simulating realistic production workload.
The recommended production configuration is set up to provide the following capacity:
- 7 simultaneous OCR jobs or a total of 10,000,000 OCR pages a year 1
- 7 simultaneous ML prediction jobs
- 2 simultaneous ML model trainings
- 5 bots
- 25 simultaneous Control Tower jobs
- 650 simultaneous (unlimited named or registered) Workspace workers
- 8 simultaneous (unlimited named or registered) Control Tower users
- 50 simultaneous Analytics users 2
For typical automation cases, the setup is capable of handling 4,000,000 transactions a year.
1 100,000 pages is a typical bundled volume. Additional pages require an extra license to be purchased.2 Five named Analytics users is a typical bundled volume. Additional users require an extra license to be purchased.
RPA
RPA Units number estimation
When planning the RPA capacity, your first step is to estimate the number of RPA Units required to handle the scope of work.
For a possible estimation approach, refer to Estimate number of RPA Units.
Horizontal scaling
The RPA component is scaled by increasing horizontally the hardware resources: VDIs or RPA servers.
Scaling to 500 bots is possible purely by adding the RPA resources. For a higher quantity of bots, multiple deployments are recommended.
Production deployment on VDI
When bots are deployed using the VDI technology, each bot has a dedicated VDI.
A typical VDI capacity is as follows:
- 2 cores
- 8 GB RAM (4 GB minimum)
- 50 GB HDD (10 GB minimum)
Production deployment on server
In case you use Windows Server, multiple bots can be deployed to a single server.
Typical hardware requirements for bots are as follows:
| Number of bots | CPU | RAM | HDD |
|---|---|---|---|
| 2 | 4 cores | 8 GB | 100 GB |
| 5* | 8 cores | 16 GB | 250 GB |
| 10* | 16 cores | 32 GB | 500 GB |
By default, only one Bot Unit is available. The second RDP connection is for the Bot Master.
* To run more than two bots on a single server, a Microsoft RDS license is required. For more details, see Activate RDS licensing for multiple bots.
Depending on the software to be automated by RPA and therefore installed on all RPA servers, you may need different amounts of RAM or HDD to perform RPA tasks by multiple concurrent bots effectively. The table above is for the minimal operational configuration and will not necessarily fit your specific case.
OCR
OCR engine allows the processing of incoming requests in parallel using multiple CPU cores. Requirements for one processor are as follows:
- 1 core
- 2 GB RAM
- 10 GB HDD
In the OCR cluster, one CPU is reserved for orchestration (Master) activities. On the rest of the CPU Cores, OCR requests can run in parallel.
workers = CPU Cores - 1
Calculate transaction throughput
You can calculate the minimal number of required OCR workers if you know the following details:
- TT (transaction time): the average time for processing a single item
- N: the number of transactions over a specified time interval
- T: the time interval in seconds
min(workers) = TT * N / T
Example:
For a one-page document, OCR processing takes 20 seconds (TT = 20). To process 30 documents (N = 30) within one minute (T = 60), you need 20 x 30 / 60 = 10 workers minimum.
Scenario A
- 10 seconds per page
- 5 pages per document
| Servers | CPU | Pages per hour | Pages per day | Pages per month | Documents per hour | Documents per day | Documents per month |
|---|---|---|---|---|---|---|---|
| 1 | 4 | 1080 | 25920 | 777600 | 216- | 5184 | 155520 |
| 1 | 8 | 2520 | 60480 | 1814400 | 504 | 12096 | 362880 |
| 1 | 16 | 5400 | 129600 | 3888000 | 1080 | 25920 | 777600 |
| 1 | 32 | 11160 | 267840 | 8035200 | 2232 | 53568 | 1607040 |
| 2 | 4 | 2160 | 51840 | 1555200 | 432 | 10368 | 311040 |
| 2 | 8 | 5040 | 120960 | 3628800 | 1008 | 24192 | 725760 |
| 2 | 16 | 10800 | 259200 | 7776000 | 2160 | 51840 | 1555200 |
| 2 | 32 | 22320 | 535680 | 16070400 | 4464 | 107136 | 3214080 |
| 4 | 4 | 4320 | 103680 | 3110400 | 864 | 20736 | 622080 |
| 4 | 8 | 10080 | 241920 | 7257600 | 2016 | 48384 | 1451520 |
| 4 | 16 | 21600 | 518400 | 15552000 | 4320 | 103680 | 3110400 |
| 4 | 32 | 44640 | 1071360 | 32140800 | 8928 | 214272 | 6428160 |
| 8 | 4 | 8640 | 207360 | 6220800 | 1728 | 41472 | 1244160 |
| 8 | 8 | 20160 | 483840 | 14515200 | 4032 | 96768 | 2903040 |
| 8 | 16 | 43200 | 1036800 | 31104000 | 8640 | 207360 | 6220800 |
| 8 | 32 | 89280 | 2142720 | 64281600 | 17856 | 428544 | 12856320 |
Scenario B
- 10 seconds per page
- 10 pages per document
| Servers | CPU | Pages per hour | Pages per day | Pages per month | Documents per hour | Documents per day | Documents per month |
|---|---|---|---|---|---|---|---|
| 1 | 4 | 1080 | 25920 | 777600 | 108 | 2592 | 77760 |
| 1 | 8 | 2520 | 60480 | 1814400 | 252 | 6048 | 181440 |
| 1 | 16 | 5400 | 129600 | 3888000 | 540 | 12960 | 388800 |
| 1 | 32 | 11160 | 267840 | 8035200 | 1116 | 26784 | 803520 |
| 2 | 4 | 2160 | 51840 | 1555200 | 216 | 5184 | 155520 |
| 2 | 8 | 5040 | 120960 | 3628800 | 504 | 12096 | 362880 |
| 2 | 16 | 10800 | 259200 | 7776000 | 1080 | 25920 | 777600 |
| 2 | 32 | 22320 | 535680 | 16070400 | 2232 | 53568 | 1607040 |
| 4 | 4 | 4320 | 103680 | 3110400 | 432 | 10368 | 311040 |
| 4 | 8 | 10080 | 241920 | 7257600 | 1008 | 24192 | 725760 |
| 4 | 16 | 21600 | 518400 | 15552000 | 2160 | 51840 | 1555200 |
| 4 | 32 | 44640 | 1071360 | 32140800 | 4464 | 107136 | 3214080 |
| 8 | 4 | 8640 | 207360 | 6220800 | 864 | 20736 | 622080 |
| 8 | 8 | 20160 | 483840 | 14515200 | 2016 | 48384 | 1451520 |
| 8 | 16 | 43200 | 1036800 | 31104000 | 4320 | 103680 | 3110400 |
| 8 | 32 | 89280 | 2142720 | 64281600 | 8928 | 214272 | 6428160 |
The following factors can affect the processing speed:
- OCR parameters, for example, the number of output document types.
- CPU clock speed.
- Document complexity.
- Efficiency of a Business Process in loading the OCR queue. Ideally, the number of documents in the queue must be 30% more than the number of workers. For example, there can be 32 workers and 40 documents in the queue.
To estimate the exact hardware requirements, run several documents through OCR on a test server and check the OCR logs for timing details.
Machine Learning
Additional system resources can be required for the following cases:
- Parallel extraction with different models
- Scheduled parallel training
- Customized system requirements for model training and prediction
You can set up a Mesos cluster on several ML servers and use all system resources of the cluster.
You can estimate exact CPU and RAM requirements during runtime. The best option is to start with the recommended production setup.
In addition, consider the following:
- The computation-intensive training or extract process is performed on demand or at intervals.
- Training can be run in a cloud using the WorkFusion infrastructure if input data can be supplied.
ML capacity planning components
Search Engine training
Running the Search Engine to produce a model requires the following resources for each field:
- 1 core
- 12 GB RAM
- 30 minutes of processing time per field
By adding more resources, you can train the Search Engine for multiple fields in parallel.
Training is executed on Agent machines. On a single Agent machine, it is possible to execute up to two parallel training jobs.
Model prediction
To use a trained (multi-field) model on an unseen document to generate ML predictions, the following resources are essential:
- 1 core
- 4 GB RAM
- 5 GB HDD
- a few seconds of the processing time (ranges from milliseconds to a minute depending on the document size) per field; prediction for a document is made sequentially, field by field.
If server resources are available, the clustering engine automatically spins up additional model instances to process multiple incoming requests from the queue concurrently.
Prediction is executed on Agent machines.
On a single Agent, one CPU and 4 GB of RAM are reserved for the orchestration and operating system activities. The remaining 7 CPU cores and 28 GB of RAM are available for job executions.
At default Worker settings, this allows executing up to 7 parallel jobs per Agent.
Advanced model resource tuning
As the amount of resources for model predictions is highly dependent on the document size, it is possible to tune resources required for an AutoML Worker based on the known document size.
Classification models
It is safe to decrease Worker CPU to 0.7 and memory to 1 GB for classification models if the document size after OCR is less than 6 KB (less than 1,000 words (tokens) per document). It allows running up to 10 parallel Workers on the same Agent.
Information Extraction models
Multi-field models for information extraction are quite CPU-intensive, so the default setting of 1 CPU is recommended. But it is possible to adjust the Worker's memory.
The document sizes after the OCR operation are estimated in the following way:
- S: 300-3,000 tokens (words) per document, 1-5 pages (~ 120 KB)
- M: 3,000-5,500 tokens (words) per document, 5-10 pages (~ 200 KB)
- L: 5,500-80,000 tokens (words) per document, 10-15 pages (~ 500 KB)
- XL: 80,000+ tokens (words) per document, 15+ pages (~ 1 MB+)
Based on the estimation, the formula for the Worker memory size is the following:
Memory in Mb = ROUNDUP( 520 * (1 + number_of_fields * 0.06) * document_size_coefficient, -2);
Here, document_size_coefficient must be taken from the following table:
| Document Size | document_size_coefficient |
|---|---|
| S | 1 |
| M | 1.35 |
| L | 1.9 |
| XL | 3.8 |
Control Tower tasks
Control Tower tasks are executed on BEP Agent machines. Those are the same machines where Machine Learning jobs are executed.
Each Worker requires the following resources:
- 0.25 core
- 1 GB RAM
A single BEP Agent machine can run up to 28 workers in parallel.
Bot Execution Platform Agents
BEP Agent machines execute Machine Learning and Control Tower tasks.
You can increase the number of BEP Agents up to 20 maximum.