Create Automation Manual Task
WorkFusion AutoML provides automation by making ML models learn from user answers provided in Manual Tasks — web forms with data and input fields. Manual Tasks are designed in Control Tower and then filled and submitted by users in WorkSpace. After a model is trained, WorkFusion creates a Cognitive Bot and routes work to it.
Select OOTB Operation
Let's start by creating and configuring a Manual Task in Control Tower.
Click Manual Tasks > Create Manual Task.
Select the type of a Task below and press Select This Operation.
Your Manual Task for the Operation has been created.
note
ALL fields that need to be automated by the model MUST be configured as required. Otherwise, the AutoQC process will fail to pick those fields.
Upload input data
Next, let's upload some data for a model to train on. This data is prepared in the Data Analyst Guide. Note that your input file should contain at least 20 records (documents) or more.
To upload your data, follow these steps:
In your Manual Task, click the Data tab, and then Upload Data.
Click Add, and then locate your CSV file, and then click Open.
Wait till the file is processed, and then click Upload.
Now that we have some data uploaded, let's configure the model.
Model not applied option
In case you don't want to automate your manual task, select Model not applied on the Configure AutoML tab.
Execute model option
For a description of the Execute model option, refer to Developing with AutoML SDK | Execute model.
Configure AutoML
To train a model and eventually generate a Cognitive Bot, link your Manual Task to AutoML and configure it.
On the Configure AutoML tab, select Train new model.
In the Machine Learning Pipelines section, select the Machine Learning Model that corresponds to your Task (for example, Information Extraction, Classification, Financial Reports Data Extraction, etc.). This model will learn from human answers and automate your Manual Task by creating a Cognitive Bot.
Select your Cognitive Bot Workflow. The Cognitive Bot will be created from this Cognitive Bot Workflow when your Manual Task is completed. You can edit the Cognitive Bot Workflow by selecting next to the Cognitive Bot Workflow drop-down menu.
Select Automation Training Set.
If you have some historical data for training the ML models, you can upload this data to the Automation Training Sets and then use it for Manual Task automation. When assigning an existing Training Set to a new Manual Task or doing a quick assessment of data for ML purposes, you can see the Training Set answer availability and its percentage using the Show Data Completeness button.
If you do not have a Training Set, select the Create new Training Set automatically option.
It is a default option. A new Training Set is created for storing Workers' answers and training ML models to automate your Task.
Optionally, click the 3-dot button to perform the following actions with the already added training set:
- View
- Rename
- Copy
- Reset Data Status
The latter sets the train_status column for all records to NEW. Used to include all records from a particular Training Set in an ML model training.
Search Engine 2.0 training configuration
Models with SE 2.0 are already optimized to produce the best results, so that you can adjust only the number of Fields Trained In Parallel. The maximum number of fields trained in parallel is calculated according to AutoML cluster capacity. The Training Configuration tab also gives an estimate of complete training time for the model.
Search Engine 1.0 training configuration
If the selected Machine Learning Model supports Search Engine 1.0, then the Training Configuration section is enabled. By default, you can see preset options for Training Configuration and the time needed to complete the training for all fields of the initial Manual Task.
Move the slider to set up your training configuration according to the complexity of the task. Moving the slider instantly re-calculates the estimated training time.
To have more control over the training setup, click Show Details. In the following picture, you can see an example of the Training Configuration section for the Information Extraction ML model.
The following parameters are available:
- Max Training Time — Maximum amount of time AutoML will use to find the best subset of components required for a single field.
- Fields Trained in Parallel — Maximum number of fields (sub-models) that can be trained in parallel on an AutoML cluster. By default, this slider is set to the optimal value based on the available cluster capacity.
note
The Fields Trained in Parallel parameter is available only for Information Extraction models and does not apply to Classification models.
- Metric — Metric used to compare the results of experiments and select the best one. Available metrics: F1, Precision, or Recall.
Values for the Max Training Time and Metric parameters are predefined and listed in the table below. In contrast, values for the Fields Trained In Parallel parameter are calculated according to the AutoML cluster capacity. There are four possible preset values ranged from High speed to High quality.
| Slider Position (Preset) | Max Training Time | Metric |
|---|---|---|
| 1 — Low Complexity | 1 hour | F1 |
| 2 | 2 hours | F1 |
| 3 | 4 hours | F1 |
| 4 — High Complexity | 10 hours | Precision |
You can select one of the preset options and it will be applied to all fields (sub-models) while training. In case you cannot find an appropriate combination of parameters from the preset options, you can provide custom parameters by clicking the Set Custom Goal link.
Using this mode you can set up all parameters manually. The parameters have the following restrictions:
- Max Training Time ranges from 1 hour to 10 hours.
- Fields Trained In Parallel is limited by the AutoML cluster capacity.
note
While calculating Fields Trained In Parallel and Complete Training Time, currently running model trainings on an AutoML cluster are not taken into account. In case one or several models are trained on an AutoML cluster, values for Fields Trained In Parallel and Complete Training Time will remain the same and will not reflect the time in a queue. Before setting the Training Configuration parameters, check other models with HPO trained on the same cluster.
Priority settings
Since there are several places for training settings configuration, there are different priorities that each place has. Priorities are listed in descending order, i.e., at the top are the settings that will be applied primarily; in the bottom are settings that will be applied last.
For Max Training Time:
- Settings provided in the Training Configuration section on the Configure AutoML tab
- Settings provided in Zookeeper in section Search Engine limiting settings
- Settings provided in Model configuration
For Fields Trained in Parallel:
- Settings provided in Training Configuration section on the Configure AutoML tab
- Settings provided in Zookeeper for Search Engine limiting settings
For Metric:
- Settings provided in Training Configuration section on the Configure AutoML tab
- Settings provided in Model configuration
Note that managing the Training Configuration section requires Manage Platform Monitor permissions for ML Engineer. Go to System Settings > Role management > Select Role Permissions > check Platform Monitor.
Automate Manual Tasks in BP
Besides automating stand-alone Manual Tasks as described above, you can also automate Manual Tasks that are steps within a Business Process.
To automate a Manual Task within a Business Process, do the following actions:
- Set AutoML parameters for this step as described here.
- Run the BP and wait until the Manual Task is completed by human workers in Workspace. Model training starts as a background process.
When the Manual Task has the AUTOMATION AVAILABLE label, proceed with these steps:
- Select minimal acceptable accuracy, and then click Apply Recommendation. See Model retraining for details. The Manual Task on the original BP Diagram is substituted with an Automation Sub-Process for this step.
note
The original BP stays unchanged, WorkFusion engine creates a new Process Definition with links to the same steps as in the original, except the automated Manual Task which is substituted with an Automation BP.
- Upload a new data batch for extraction, and then launch this new BP. You can expand the Sub-Process and view the automation progress.
tip
You can automate several Manual steps in a BP the same way.