Edit

Infrastructure Capacity Planning

Summary

Infrastructure capacity planning is the process of determining the hardware and software resources required to meet current processing demand as well as over some future period of time. A common concern of enterprises is whether the required resources are in place to handle an increase in users or number of transactions.

Capacity Planning is a part of a wider field of Capacity Management. Capacity Management is concerned with ensuring that cost-effective capacity exists at all times which meets or exceeds the agreed needs of the business as established in Service Level Agreements. Capacity is defined as the maximum throughput a service, system, or device can handle.

Standard build

Refer to System Requirements that summarize the recommended standard specification for the environment to run WorkFusion SPA.

The recommended production configuration is set up to handle the following¹:

• 40 active business processes
  • total of 4 000 000 transactions a year
• 12 000 000 OCR pages a year ²
• 30 simultaneous (300 deployed models) ML prediction jobs
• 1 simultaneous HPO ML model training
• 5 RPA nodes ³
• 50 simultaneous (unlimited of named/registered) WorkSpace workers
• 8 simultaneous (unlimited of named/registered) Control Tower users
• 50 simultaneous Analytics users ⁴

¹ – There are a number of factors that can impact performance and scalability of the software. Some of the important system variables include business processes design, server configuration, infrastructure tuning, data environment, compute capacity, and networking. The above numbers were established in a closed network lab on automated load tests mimicking realistic production workload.

² – 100K pages is typical bundled volume. Additional pages require an extra license to be purchased.

³ – The easiest component to scale. Keep reading.

⁴ – 5 named Analytics users is typical bundled volume. Additional users require an extra license to be purchased.

RPA

First step in planning the RPA capacity is estimating number of RPA nodes required to handle the work.

Refer to the following pages for approach to estimate it:

Number of bots estimation

Horizontal scaling

The RPA scaling is done by horizontally scaling the hardware resources (VDIs or RPA servers) executing RPA tasks.

Scaling to 100 RPA nodes is possible purely by adding the RPA resources.

To scale beyond 100 - Application server upgrade may be required. For example upgrading CPU on Application server to 32 cores will allow horizontal scaling of the RPA component up to 500 RPA nodes.

Minimal system requirement for single RPA node

Single RPA node is operational on the following system specs:

• 1 core
• 2 GB RAM
• 10 GB HDD

Production deployment on VDI

In case of RPA nodes deployments on VDI technology. Each RPA node has a dedicated VDI.

Typical VDI capacity:

• 2 cores
• 8 GB RAM
• 50 GB HDD

Production deployment on server

In case of RPA nodes deployments on Windows Server. Multiple RPA nodes can be deployed to a single server.

Typical for 5 RPA nodes:

• 8 cores
• 16 GB RAM
• 250 GB HDD

Typical for 10 RPA nodes:

• 16 cores
• 32 GB RAM
• 500 GB HDD

For this type of deployment you need Terminal Services license.

OCR

OCR engine allows processing of incoming requests in parallel using multiple CPU cores.
Requirements for 1 processor:

• 1 core
• 4 GB RAM
• 10 GB HDD

ABBYY License comes with 4 enabled cores by default, but can be extended to 8 with >1M pages license

Transaction throughput calculation

It's possible to calculate number of desired OCR Workers or RPA Nodes knowing the following details:

• transaction time (average time for processing single item) - TT
• number of transactions per time interval - N
• time interval - T

Workers/Nodes = TT * N / T

Example: OCR processing takes 20 seconds (TT = 20) for 1-page document. If we want to process 30 documents (N = 30) within 1 minute (T = 60) - we need 20 * 30 / 60 = 10 workers

Scenario A

• 10 secons per page
• 5 pages per document

We can't preview this file. You'll have to download the file to view it.

Scenario B

• 10 secons per page
• 10 pages per document

We can't preview this file. You'll have to download the file to view it.

Factors which might affect processing speed:

• OCR parameters (example - number of output document types)
• CPU clock speed
• Document complexity
• How efficiently Business Process loads OCR queue (its best when there is always additional 30% documents in queue comparing to number of workers, example - 32 workers and 40 documents in queue. This can be achieved via setting high number of threads in Bot Source)

In order to estimate exact hardware requirements - it's better to run several documents through OCR on test server and check OCR logs for timing details

Machine Learning

Additional system resources might be required for the following cases:

• Parallel extraction with different models
• Parallel training is scheduled

It is possible to setup Mesos cluster on several ML servers and utilize all system resources of the cluster.

Exact CPU/RAM requirements can be estimated during runtime - but always start with recommended production setup.

Additional things to consider:

• Compute intensive training or extract process is done on demand or with intervals.
• Training can be run in cloud using WorkFusion infrastructure if input data can be supplied.

ML capacity planning building blocks

HPO training

Running HPO to produce metamodel takes the following resources for each field:

• 32 cores
• 100 GB RAM
• 10 hours of processing time per every field

Adding more resources lets running HPO training for multiple fields in parallel

Metamodel re-training

Re-training metamodel takes the following resources for each field:

• 4 cores
• 32 GB RAM
• 1 hour of processing time per every field

Adding more resources lets running metamodel re-training for multiple fields in parallel

Model prediction

Use a trained model (multi-field) on an unseen document to generate ML predictions takes:

• 1 core
• 4-6 GB RAM
• 5 GB HDD
• a few seconds of the processing time (ranges from milliseconds to a minute depending on the document size) per field (prediction for a document is done sequncially, field by field)

If server resources available the clustering engine automatically spins up additional model instances to process multiple incoming requests from the queue concurrently.

Requirements and calculations

The requirements may differ based on the training strategy selected and hardware requirements.

Calculator

The studies from this page has been collected in the below calculator.

The best use of the calculator and this page is to produce a high level estimate of the capacity required and hardware cost, especially for low and mid scale deployments.

For optimal resources utilization and processing speed it's beneficial to build an independent capacity planning framework accounting for the infrastructure features, previous studies, load testing results, performance optimizations applied.

For high volume large scale deployments it’s recommended to do additional analysis, have dedicated stages for high scale design, implementation, testing and load testing, configuration and implementation optimizations.

capacity planning

• Latest version: Infrastructure Capacity Planning v5.0.1.xlsx
• Older versions: HW_Capacity Planning_v3.xlsx