10.0

Edit

View AutoML SDK flow

The article features a high-level overview of how you can develop a Machine Learning (ML) model with AutoML SDK and publish it to Control Tower (CT) for further use in a Business Process (BP).

Build model

The basic flow to deliver an ML model is as described below:

  1. Create an AutoML SDK project.

  2. Implement a generic model.

  3. Improve the resulting ML model, if needed.

Create AutoML SDK project

To create an AutoML SDK project, follow the instruction below:

  1. Make sure you have access to an AutoML SDK installation—either from the WorkFusion Maven repository or your local on-premise one.

  2. Configure a local environment as detailed in the guide.

  3. Configure a project as described in the guide.

Implement generic model

Now that you have a ready-to-go AutoML SDK project, implement a generic model as described below:

  1. Configure ModelTrainingRunner to extract required fields in accordance with the Train model locally guide.
  2. Run ModelTrainingRunner in accordance with the instruction.
  3. In the output/model-results-stats folder, check the model's statistics.
  4. If you have no improvements to make, proceed to publish and execute the model. Otherwise, read the Improve model section.

Improve model

In some cases, you may need to improve one or more fields in an already existing trained model. Retraining the entire model is a time-consuming operation. To save time, you can improve only the required field and replace the artifact folder containing the old field with a new one containing updates.

To improve a model, follow the steps below:

  1. Make a separate copy of your AutoML SDK project.

  2. Import one or more fixed serialized configurations to the AutoML SDK configuration file for all fields. In this case, AutoML skips the smart selection of components, which significantly reduces the time of further retraining.

  3. Comment all fields in your ModelTrainingRunner, uncommenting the ones that need to be improved.

    tip
    • If a model extracts fields but they are formatted incorrectly, consider adding Post-Processors to your AutoML SDK configuration.
    • If a model extracts wrong fields, consider adding Feature Extractors and Annotators to your AutoML SDK configuration.

  4. Run ModelTrainingRunner with the updated AutoML SDK configuration.

If you're satisfied with the result, proceed to publish and execute the model.

Reuse existing model

The development flow reusing an existing model is almost identical to that when a model is created. Instead of configuring a new project, import an existing one.

To start developing based on an existing model, follow the steps below:

  1. Make sure you have access to an AutoML SDK installation—either from WorkFusion Maven repository or your local on-premise repository.
  2. Configure the local environment as detailed in the guide.
  3. Import your existing AutoML SDK project to WorkFusion Studio.

Then, either use the model as is or make improvements.

Explore resulting project structure

Below is an example of the resulting structure for a trained model project:

ProjectForTest
├── /data
├── /deploy
|    └── /ProjectForTest
|        └── /1.0
|            ├── /config
|            ├── /lib
|            └── model.description
├── /process
├── /results
|    ├── /extract
|    └── /training
|        ├── /input
|        ├── /output
|        ├── /process
|        └── /work
├── /src   
├── /target  
└── pom.xml

Publish and execute

After you successfully implement a model or reuse an existing one, your next step is to publish it for production purposes. For that, copy the resulting artifacts to a remote file storage and make it available on a remote Control Tower.

There are two options to publish your model:

  • Publish model configuration only. In this case, training is performed on a remote Control Tower (CT) instance. After that, the model becomes available for execution.

  • Publish a trained model with all training artifacts. This scenario allows you to avoid remote model training and use the model for extraction or classification immediately.

Publish model configuration

To publish the model to an S3 or on-premise storage, follow the steps below:

  1. Run maven install using the pom.xml file in the root of the project to generate the /deploy folder.

  2. Copy the /deploy/ProjectForTest folder to s3://vds-models/.

  3. Add a new record with the ProjectForTest:1.0 value to the s3://vds-models/models./ file. This makes the model visible in Control Tower configured to work with the S3 bucket.

  4. Check the model availability in Control Tower:

    • On the Manual Tasks list, select a Manual Task with the draft status or create one. Navigate to the Configure AutoML tab, and select Train new model or Execute new model. In the Machine learning drop-down list, look for your model, paying attention to its title and version.

    • Via AutoML SDK API, send a listHyperModels request.

Configure model training

After publishing, you can configure a training in Control Tower for the new model:

  1. On the Manual Tasks list, select a Manual Task or create one.

  2. Click the Configure AutoML tab and select Train new model.

  3. In the Machine learning model drop-down box, select the newly published ML model.

In the publishing case described above, you can use only the model's configuration. Model training is to be run on the remote Control Tower instance from scratch.

Publish trained model

To avoid remote retraining and use the model for extraction or classification immediately, you can publish a trained model directly to S3 as described below:

  1. Publish the model configuration as described above.
  2. Copy the /training/output folder to s3://vds-resources/.
note

The name of the S3 folder (in the sample project above, /training) must be exactly the same as in your AutoML SDK project in WorkFusion Studio.

Make sure there are no backslashes (\) in the path to parameters.json. If you find ones, replace them with a regular slash (/).

Execute model

important

Large hyper models and trained models, exceeding 500 MB, can cause high IOPS (high iowait) when Workers are launched.

To use the new model in a Business Process, add it to an AutoML Business Process in Control Tower. For the purpose, you can create a Manual Task or reuse an existing one.

  1. On the Manual Tasks list, select a newly created or an existing Manual Task.

  2. Click the Manual Task name to navigate to the Configure AutoML tab and fill in all fields, paying attention to the rules below:

    • The Machine Learning Model drop-down list fetches a title from the model.description file of your AutoML SDK project. Make sure to give the model a meaningful name before you deploy it.

    • Training Model ID is the name of the folder from your project in WorkFusion Studio. In the sample project above, it is the /training folder copied to s3://vds-resources/.

    • Group ID is combined-results for Information Extraction.

  3. Click Save.

  4. Go to the AutoML tab and click Activate Cognitive Bot. This opens the Automation BP wizard.

← Configure modelConfigure local environment →