Train model locally

A model generated from an Archetype contains a local runner that allows training the configured model. The runner is located at ../run/ModelTrainingRunner.java.

The ModelTrainingRunner class contains a single method. For Information Extraction, provide a path to the dataset and a working directory and specify a FieldInfo field.

import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;

import com.workfusion.vds.sdk.api.nlp.configuration.FieldInfo;
import com.workfusion.vds.sdk.api.nlp.configuration.FieldType;
import com.workfusion.vds.sdk.run.ModelRunner;
import com.workfusion.vds.sdk.run.config.LocalTrainingConfiguration;

public class ModelTrainingRunner {

    public static void main(String[] args) throws Exception {
        System.setProperty("WORKFLOW_LOG_FOLDER", "./logs/");

        //Configure input and output
        Path inputDirPath = Paths.get("TODO add path to folder with training set");
        Path outputDirPath = Paths.get("TODO add path to output folder");

        //Configure the fields according to your use case
        List<FieldInfo> fields = new ArrayList<>();
        fields.add(new FieldInfo.Builder("invoice_number")
            .type(FieldType.INVOICE_TYPE)
            .build());
        fields.add(new FieldInfo.Builder("total_amount")
            .type(FieldType.PRICE)
            .build());
        //Add parameters, if needed
        Map<String, Object> parameters = new HashMap<>();

        LocalTrainingConfiguration configuration = LocalTrainingConfiguration.builder()
                .inputDir(inputDirPath)
                .outputDir(outputDirPath)
                .fields(fields)
                .parameters(parameters)
                .build();
        //Run your model specified by the Hypermodel class and configuration
        ModelRunner.run(TrainingIeModel.class, configuration);
    }
}

When training is finished, the resulting model is available at workingDir/training/output/model. The path is passed to the execution runner to check the model quality.

Run ModelTrainingRunner

To make the runner work, provide the following parameters:

• inputDirPath is the path to the folder with training set files.
• outputDirPath is the path to the output folder where trained model artifacts are stored.
• FieldInfo fields represent the configuration of an Information Extraction model (if you use one).
• ModelRunner.run() is your custom model class.
• Optional: add additional parameters, for example, additional training configuration, and so on.
• Optional: add additional inputs, such as dictionaries, reference data, and so on. Use the addResource(targetFolder, sourceFile/Folder) method.

See sample ModelTrainingRunner configuration for invoice_number single field and corresponding type

import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;

import com.workfusion.vds.sdk.api.nlp.configuration.FieldInfo;
import com.workfusion.vds.sdk.api.nlp.configuration.FieldType;
import com.workfusion.vds.sdk.run.ModelRunner;
import com.workfusion.vds.sdk.run.config.LocalTrainingConfiguration;

public class ModelTrainingRunner {

    public static void main(String[] args) throws Exception {
        System.setProperty("WORKFLOW_LOG_FOLDER", "./logs/");

        Path inputDirPath = Paths.get("/home/ie-example/input");
        Path outputDirPath = Paths.get("/home/ie-example/output");

        List<FieldInfo> fields = new ArrayList<>();
        fields.add(new FieldInfo.Builder("invoice_number")
                .type(FieldType.INVOICE_NUMBER)
                .required(true)
                .multiValue(false)
                .build());

        Map<String, Object> parameters = new HashMap<>();

        LocalTrainingConfiguration configuration = LocalTrainingConfiguration.builder()
                .inputDir(inputDirPath)
                .outputDir(outputDirPath)
                .fields(fields)
                .parameters(parameters)
                .addResource("dictionary", Paths.get("home/ie-example/dictionary/InvoiceNumbers.csv"))
                .build();

        ModelRunner.run(ExampleIeModel.class, configuration);
    }
}

See sample ModelTrainingRunner configuration for invoice_number group field and corresponding type

import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;

import com.workfusion.vds.sdk.api.nlp.configuration.FieldInfo;
import com.workfusion.vds.sdk.api.nlp.configuration.FieldType;
import com.workfusion.vds.sdk.run.ModelRunner;
import com.workfusion.vds.sdk.run.config.LocalTrainingConfiguration;

public class ModelTrainingRunner {

    public static void main(String[] args) throws Exception {
        System.setProperty("WORKFLOW_LOG_FOLDER", "./logs/");

        Path inputDirPath = Paths.get("/home/ie-example/input");
        Path outputDirPath = Paths.get("/home/ie-example/output");


        List<FieldInfo> fields = new ArrayList<>();
        fields.add(new FieldInfo.Builder("invoice_group").type(FieldType.GROUP)
                    .child(new FieldInfo.Builder("invoice_number")
                        .type(FieldType.INVOICE_TYPE)
                        .build())
                    .child(new FieldInfo.Builder("invoice_date")
                        .type(FieldType.INVOICE_DATE)
                        .build())
                .build());

        Map<String, Object> parameters = new HashMap<>();

        LocalTrainingConfiguration configuration = LocalTrainingConfiguration.builder()
                .inputDir(inputDirPath)
                .outputDir(outputDirPath)
                .fields(fields)
                .parameters(parameters)
                .addResource("dictionary", Paths.get("home/ie-example/dictionary/InvoiceNumbers.csv"))
                .build();

        ModelRunner.run(ExampleIeModel.class, configuration);
    }
}

View model training results

After successful model training, the output folder contains a training sub-folder with the following folders:

• output is the folder with the model output, for example, statistics or trained model artifacts.
• process is the folder with the model training details, for example, status files or training logs.
• work is the working folder for the model training, for example, feature or temporary files.

The trained model artifact is added to the <%output_folder%>/training/output/model folder.

Apply workaround for OutOfMemoryException

When a training set contains huge amounts of data, local training can throw OutOfMemoryException. To avoid this, add the following parameter to ModelTrainingRunner:

parameters.put(ConfigurationConstants.PARAM_MAX_FIELDS_IN_PARALLEL, 1)