ML training strategies
HPO vs Metamodel retraining
In WorkFusion training of new ML models is done in two stages:
- Hyper Parameter Optimization (HPO). Traversal of hyperspace (defined by the Hypermodel) to find the best Metamodel for the supplied Training Set.
- Model training. Metamodel usage to train a model on the supplied Training Set.
The first stage is the most resource heavy and is advised for automatic discovery of the best features and parameters of the Metamodel for this type of use case.
Further improvements of the ML performance for the same use case may not necessary need full HPO re-traversal, but rather simple retraining of the Metamodel on the updated data set.
Training strategies
The main strategies for training ML models are:
| Strategy # | HPO | Model retraining |
|---|---|---|
| 1 | CLOUD | CLOUD |
| 2 | CLOUD | DEV |
| 3 | DEV | DEV |
| 4 | DEV | PROD |
- CLOUD: Scalable and high-performing WorkFusion VDS cloud. Best for complex use cases with lots of data.
- DEV: environment different from the business critical (operational) environment. Best for on-premise experimenting with hypermodels and configurations.
- PROD: production environment used for executing automation business processes targeted for operations optimizations.
Decision on applying one or another strategy usually is based on the following factors:
- Target Automation Rate (potential automation rate decreases with strategy number increasing)
- Server Capacity (hardware requirements grow with strategy number increasing)
- QA and UAT requirements (strategies 1-3 have QA/UAT before applying the model in prod, strategy 4 applies new models automatically)
- Data Privacy and Control (strategies 1-3 require the training data to be moved to the training environment, strategy 4 trains new models right in place)
On premise HPO training
This section covers the strategy # 3 and # 4 where the heavy HPO process is performed on DEV environment, and once a Metamodel retrieved - further improvements are achieved in either DEV or PROD environments. DEV would be as a rule used in case additional change management control (e.g. QA/UAT) is required prior to enabling the new model in production.
Prerequisites to build model
- Production data or close to production data (should not be synthetic) in the Training Set. For example, real invoices, SSI documents, etc.
- Data tagged by subject-matter experts (SME) – gold dataset without errors.
- Generic HPO Hypermodel available on DEV S3 storage.
Phase 1: HPO Training in DEV
- Get production tagged data (~1500 documents) and put it on DEV S3 storage.
- Split production tagged data to Training Set (80%) and Test Set (20%).
- Run training set through HPO Hypermodel on DEV > HPO will produce best Trained Model.
- Run Test Set through the Trained Model and save extraction statistics.
- Pick up fixed configuration from Trained Model and build Metamodel (fixed configuration hypermodel) using ML SDK > save Metamodel on DEV S3.
- Migrate Metamodel and Trained Model to PROD S3.
Phase 2: Model Extraction PROD
- In a Manual Task on the PROD plug production Training Set with Metamodel and with Trained Model, set up retraining new documents threshold, for example, 100 > extract new data with the same quality as on DEV.
- Each document reviewed by a human is added to Training Set.
- Once the configured amount (threshold) of new documents is available in Training Set (threshold met) > Retraining will start in background.
Phase 3: Model (re)training
Model retraining can be executed either in DEV or in PROD environment, depending on your architecture and security requirements.
PROD
Metamodel will produce New Trained Model each time new documents, for example, 100 (threshold), are available in Training Set.
New Trained Model will automatically replace Old Trained Model in case it demonstrates better automation performance.
DEV
Alternatively, model retraining can be executed in the DEV environment using the following steps:
Model Extraction step is the same as in previous method.
After a Training Set is collected in PROD, these data are migrated to DEV.
Model retraining is executed in DEV environment.
After retrain, a new Trained Model is migrated to PROD S3 and is used in subsequent extractions.
