Optimize Business Process data
Every BP step accepts input data and produces output data. The BP step output data is then copied to the input of the next BP step.
Generally, the amount of the data written to the database can be estimated as follows:
| Parameter | Description |
|---|---|
| S | The number of BP stateful steps |
| T | The number of transactions processed by the BP |
| V | The amount of data passed through the BP steps |
Estimate of stored data = S T V * 4
Below is a calculation example for the following reference conditions:
- A BP has 20 steps.
- 100,000 transactions are processed per day.
- Every transaction has several data points:
- ~128 KB input data—a few-page document: a greyscale PDF or an image, an OCR result, a tagging or labeling result.
- ~4 KB in total for intermediate data points generated by the BP logic.
Estimate of stored data = S T V 4 = 20 100,000 (128,000 B + 4,000 B) 4 = 1,056 GB per day
Reduce data volume handled by Business Processes
There are several ways how you can reduce the data passing through a Business Process (BP):
Disable snapshots generation by default (deprecated from v10.3). After that, you can still generate snapshots on demand. For more details, refer to Optimize disk load.
Store big data points (for instance, documents) in a file storage, for exanple, S3 coming with Work.AI out of the box.
Example (based on the calculation example above): instead of passing the entire file content through all the steps, pass a document link. As a result, the ~128 KB input document is replaced with ~200 B link to the input document.
Estimate of stored data = S T V 4 = 20 100,000 (200 B + 4,000 B) 4 = 33.6 GB per day
Avoid passing unnecessary data points between steps. There is no need to pass all data to every step to the very end of a BP. Pass only what is actually needed on the next step. Always use
<export include-original-data="false">and include required data points only.Example (based on the calculation example above): instead of passing ~4 KB of all data points, pass only ~1 KB of required data points.
Estimate of stored data = S T V 4 = 20 100,000 (200 B + 1,000 B) 4 = 9.6 GB per day
Apply the stateless execution feature for Bot Tasks, when possible. This way, step data is stored in memory without being written to the database.
Example (based on the calculation example above): instead of saving data for every step, mark most of them as stateless. For example, let's assume only 5 out of 20 steps remain stateful.
Estimate of stored data = S T V 4 = 5 100,000 (200 B + 1,000 B) 4 = 2.4 GB per day
Reclaim disk space on DB server after data purging
The space freed as a result of data purging operations is not released to the OS file system. Instead, the freed space becomes available for new data inserts.
Optimize disk load
The automatic calculation option is deprecated starting from v10.3.
To regulate data load, you can enable or disable heavy statistical calculations. For that, go to System settings > System preferences > Schedule Settings and select or deselect the automatic calculation options.
The options are as follows:
- Automatically Generate Snapshot After Final BP step
- Automatically Generate Snapshot After Manual Task Completion
If checked, the above options can significantly increase instance workload.
Gather logs for data purging
To start tracking of data purging, enable DEBUG for the jobber logger in ZooKeeper:
$ cd /opt/workfusion/zookeeper/bin
$ ./zkCli.sh
create /config/control-tower/log/com.freedomoss.jobber DEBUG
For more information, refer to Change log level for components.
In case none of the solutions help, or you do not find the appropriate guide, gather the following logs from the Master server and send them to the WorkFusion Support Team for further analysis:
/opt/workfusion/logs/workfusion/control-tower.log.json