Edit

Test model execution locally

Prepare data and run test

To check the model quality, prepare a test set of documents and run execution. ModelEvaluationRunner.java contains logic similar to that of the training runner. Once the execution is finished, model results are available in the output directory. Statistics are located in /outputDir/statistics.csv and contain information about what actually was extracted by the model against what was expected.

Once you develop and train a model, run extraction locally to get extraction statistics. Using the extraction statistics, you can analyze the model quality and decide whether you need to improve it.

For that purpose, use the ModelExecutionRunner class generated by the Archetype. Here you can find the main method where you provide the following paths:

• trained model folder or the model folder from the output
• dataset folder
• output folder

import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.HashMap;
import java.util.Map;

import com.workfusion.vds.sdk.run.ModelRunner;
import com.workfusion.vds.sdk.run.config.LocalExecutionConfiguration;

public class ModelExecutionRunner {

    public static void main(String[] args) throws Exception {

        Path trainedModelPath = Paths.get("/home/post-processing/trained-model/");
        Path inputFolderPath = Paths.get("/home/post-processing/input/");
        Path outputFolderPath = Paths.get("/home/post-processing/output/");

        Map<String, Object> parameters = new HashMap<>();

        LocalExecutionConfiguration configuration = LocalExecutionConfiguration.builder()
                .inputDir(inputFolderPath)
                .outputDir(outputFolderPath)
                .trainedModelDir(trainedModelPath)
                .parameters(parameters)
                .build();

        ModelRunner.run(ExampleIeModel.class, configuration);
    }

}

warning

If you use Windows on your machine, mind the following prerequisites:

• Install your Java, IDE, and a working folder on the same logical drive.
• Do not add the drive name to the path. Use /home/data-set/ instead of C:/home/data-set/.

After execution, you get an output folder with the following structure:

• model-result contains JSON files with model extraction results.
• processing-result contains a JSON file where post-processing results are stored.
• statistics.csv is a file with statistics.

Inside the ModelRunner class, you can find an instance of ProcessingRunner. This is the main class that executes extraction and post-processing logic on your local machine.

The dataset folder contains a set of separate files (documents). However, if you have a CSV file only, for example, a snapshot from a Manual Task, instead of the run method, call runFromCsv, providing a path to the CSV file and the name of the column comprising the document content, for example, HTML, XML, or TXT. You can also use the TrainingSetUtils utility class that provides useful methods to convert a dataset from or to CSV and TXT files.

Analyze statistics

Let's look closer at the statistics.csv file. Using the file, you can analyze the model extraction results and post-processing output.

For each file in a dataset, there is a separate line. For each field in a dataset document, there are seven columns, where <%field\_name%> is the corresponding field name, for example, invoice_amount.

Column name	Description
<%field_name%>_gold	Value from the dataset file.
<%field_name%>_extracted	Value extracted by the trained model.
<%field_name%>_post_processed	Value after post-processing.
<%field_name%>_model_score	Score provided by the model during extraction.
<%field_name%>_final_score	Score changed by post-processing, the same as the model score if not changed.
<%field_name%>_error_type	Error type. Possible values: TP, TN, FP, FN.
<%field_name%>_reason	Reason why an error occurred, if any. Possible values: EXTRACT_MODEL, EXTRACT_RULES, FAIL_MODEL, FAIL_RULES.

Once you update the model (Annotators, Feature Extractors, or Post-Processing), train the model again, run execution, and analyze the updated statistics.

Train model with limited resources

In some cases, you may need to train a model with limited CPU resources and less than 8 GB RAM. In this case, prepare the ModelTrainingRunner class to organize training field by field, nested into a try-catch block. Mind that the approach takes more time.

package ru.sberbank.run;

import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;

import com.workfusion.nlp.uima.pipeline.constants.ConfigurationConstants;
import com.workfusion.vds.sdk.api.nlp.configuration.FieldInfo;
import com.workfusion.vds.sdk.api.nlp.configuration.FieldType;
import com.workfusion.vds.sdk.run.ModelRunner;
import com.workfusion.vds.sdk.run.config.LocalTrainingConfiguration;

import ru.sberbank.model.RussianIE;

public class ModelTrainingRunner {

    public static void main(String[] args) throws Exception {
        System.setProperty("WORKFLOW_LOG_FOLDER", "./logs/");
        System.setProperty("EVAL_WORKING_DIR_IS_UNDEFINED", "./eval/");

        List<FieldInfo> fields = new ArrayList<>();

        Path inputDirPath = Paths.get("D:\\generic\\input");
        String outputPathFolder = "D:\\dev\\train\\generic";
        LocalTrainingConfiguration configuration = null;
        String fieldName = "";
        FieldType type = FieldType.FREE_TEXT;
        Map<String, Object> parameters = new HashMap<>();
        parameters.put(ConfigurationConstants.HPO_TIME_LIMIT, 60 * 60 * 4);

        try {
            // train with the Address answer type
            fieldName = "full_address";
            type = FieldType.ADDRESS;
            fields.add(new FieldInfo.Builder(fieldName).type(type).build());
            configuration = LocalTrainingConfiguration.builder().inputDir(inputDirPath).outputDir(Paths.get(outputPathFolder + "\\model_" + "_" + fieldName))   .fields(fields).parameters(parameters).build();
            ModelRunner.run(RussianIE.class, configuration);
            fields.clear();
        } catch (Exception ex) {
            System.out.println(ex);
        }

        try {
            // train the same field with the Free Text answer type
            fieldName = "full_address";
            type = FieldType.FREE_TEXT;
            fields.add(new FieldInfo.Builder(fieldName).type(type).build());
            configuration = LocalTrainingConfiguration.builder().inputDir(inputDirPath).outputDir(Paths.get(outputPathFolder + "\\model_" + type + "_" + fieldName)).fields(fields).parameters(parameters).build();
            ModelRunner.run(RussianIE.class, configuration);
            fields.clear();
        } catch (Exception ex) {
            System.out.println(ex);
        }

        try {
            // train with the Number answer type for the Total amount field
            fieldName = "total_amount";
            type = FieldType.NUMBER;
            fields.add(new FieldInfo.Builder(fieldName).type(type).build());
            configuration = LocalTrainingConfiguration.builder().inputDir(inputDirPath).outputDir(Paths.get(outputPathFolder + "\\model_" + type + "_" + fieldName)).fields(fields).parameters(parameters).build();
            ModelRunner.run(RussianIE.class, configuration);
            fields.clear();
        } catch (Exception ex) {
            System.out.println(ex);
        }
    }
}

Customize logging levels for local run

For debug purposes, during local execution or training, you can change the logging level for any logger. You can use the SpringBoot-like approach: apply JVM parameters with the logging.level prefix. See the example below:

-Dlogging.level.root=DEBUG
-Dlogging.level.com.workfusion=WARN
-Dlogging.level.my.custom.packages=TRACE