Statistics

To better understand the model results, AutoML SDK provides comprehensive statistics for further analysis and model tuning.

For statistics calculation, AutoML SDK applies a standard MapReduce approach.

1. Statistics calculator processes one document with gold and extracted fields at a time, and then creates a data transfer object (DTO) — a lightweight representation of statistics for the processed document.
2. Statistics is calculated based on a collection of DTOs created previously.

With a number of interfaces for custom calculations we also provide a basic reference implementation of statistics calculator for Information Extraction and Classification. See the following sections for details.

Visual representation of the calculated statistics is produced via a set of OOTB Printers. Although, ML Engineers can create their own printers and plug them into the reference implementation of statistics calculator to generate various types of reports: CSV, Excel document, etc.

Main interfaces

StatisticsCalculator<I extends Document, O extends DocumentResult> is the main interface for statistics calculation and should be as well implemented for custom statistics calculation.

O process(I document)

O process(I document) gets Document as an input parameter and returns a DTO which extends DocumentResult. ML Engineer manually implements mapping from Document to DocumentResult.

Method is called for each document in a training set.

void calculate(Collection<O> results)

With void calculate(Collection<O> results), the input parameter is a collection of DocumentResults calculated during the above-mentioned Process method call. ML Engineer implements the main statistics calculator and execution of the provided Printers.

The method is called only once in the end of execution.

DocumentResult is an interface for DTO returned by the Process method during the mapping of Document to its statistics result.

String getDocumentId()

String getDocumentId() returns ID of the Document for which the results were calculated.

Information Extraction

IeStatisticsCalculator is an implementation for StatisticsCalculator in case of Information Extraction. It accepts IeDocument as an input parameter and outputs IeDocumentResult.

Using the Enable and Disable methods of the calculator you can enable or disable the following three different evaluation types:

• Value based (enabled by default). Gold and extracted results evaluated by the data-value attribute of the corresponding Field and GoldField.
• Position based. Gold and extracted results evaluated by the begin and end position of the corresponding Field and GoldField.
• Text based. Gold and extracted results evaluated by the extracted text of the corresponding Field and GoldField.

See StatisticsEvaluationType for more details.

The Calculator produces statistics based on the provided strategies. It maps the extracted result of all Fields and gold result of all GoldFields for each FieldInfo to create an object-based DTOs structure representing the result of mapping.

IeDocumentResult contains information about the mapping result of extracted and gold fields. This information is represented as DTO to store all the needed data from the corresponding Field/GoldField with a corresponding result type.

note

StatisticCalculator transforms the Field or GoldField data into a lightweight DTO that is included into DocumentResult to prevent memory issues.

Method	Description
Collection<GroupFieldResult> getGroupResults()	Returns a mapping results for group fields. Group field is a field represented as a collection of children fields.
Collection<FieldResult> getResults()	Returns mapping results between Field and GoldField aggregated into DTO.

Classification

ClassificationStatisticsCalculator is an implementation of StatisticsCalculator in case of Classification. It accepts ClassificationDocument as an input parameter and outputs ClassificationDocumentResult.

ClassificationDocumentResult contains information about the mapping result between the extracted Label and GoldLabel. This information is represented as DTOs to store data from the corresponding Label and GoldLabel.

Method	Description
Collection<LabelResult> getLabels()	Returns a mapping result for Label and GoldLabel classification.

Printers

Reference implementation contains a basic interface for all printers. Any number of printers can be plugged into the corresponding StatisticsCalculator.

StatisticsPrinter<D extends DocumentResult> is an interface for statistics printer. It should be implemented to print statistics in a custom format.

Method	Description
void print(Collection<D> results, Path output)	Input parameters: collection of statistics DTOs and a path to a target folder. By default, this method is called for each evaluation type and in each statistics calculator where it's plugged in. This method is called during the statistics calculation.

IeGoldVsExtractedCSVPrinter

IeGoldVsExtractedCSVPrinter provides statistics for Information Extraction and represents the mapping result between the gold and model extracted values.

• Input: collection of IeDocumentResult.
• Output: mapping result between each Gold and Extracted field.

This printer can be configured to store results for all field names into one file (enabled by default), or into separate files where each file corresponds to one field name (set filePerField(true)). The result will be stored in gold-vs-extracted.csv or fieldName-gold-vs-extracted.csv where fieldName is a name of field when filePerField is set to true.

• FILE_NAME: a name of document file from training/evaluation set.
• FIELD_NAME: a name of field.
• GOLD: gold value of field. Value of this column depends on the evaluation type.
• EXTRACTED: extracted value of field. Value of this column depends on the evaluation type.
• RESULT: type of mapping between the Gold and Extracted value. It can have the following values:
  • TP (True Positive) means that Gold and Extracted values are identical.
  • FP (False Positive) means that the field was extracted, but does not map to any Gold value.
  • FP, FN (False Positive, False Negative) means that both Gold and Extracted values are present, but differ.
  • FN (False Negative) means that the Gold value is present but does not map to any Extracted field.
• REASON represents information about what caused the RESULT. It can have the following values:
  • EXTRACT_MODEL means that Gold and Extracted values are identical, and were extracted by a model. Extracted by model means that value and text in extracted field are identical.
  • EXTRACT_RULES means that Gold and Extracted values are identical, and were extracted by rules. Extracted by rules means that value and text in extracted field are different.
  • FAIL_MODEL means that Gold and Extracted values are different, and failed by a model. Failed by model means that value and text in extracted field are identical.
  • FAIL_RULES means that Gold and Extracted values are identical, and failed by rules. Failed by rules means that value and text in extracted field are different.

IeGoldVsExtractedPerGroupCSVPrinter

IeGoldVsExtractedPerGroupCSVPrinter provides statistics for Information Extraction and represents the mapping result between the gold and model extracted group field values.

• Input: collection of IeDocumentResult.
• Output: mapping result between each Gold and Extracted group fields. Group field is a collection of multi-value child fields associated via the tabNumber attribute.

This printer can be configured to store results for all field names into one file (enabled by default), or into separate files where each file created for one field name (set filePerField(true)). The result will be written into group-gold-vs-extracted.csv or fieldName-group-gold-vs-extracted.csv where fieldName is a name of field when filePerField is set to true.

• FILE_NAME: name of a document file from training or evaluation set.
• GROUP_NAME: name of a group field.
• GOLD_TABNUMBER: value of the tabnumber attribute of all child fields of gold group field.
• EXTRACTED_TABNUMBER: value of the tabnumber attribute of all child fields in the extracted group field.
• EXTRACTED: extracted value of field. Value of this column depends on the evaluation type.
• RESULT: type of mapping between the Gold and Extracted value. It can have the following values:
  • TP (True Positive) means that Gold and Extracted values are identical.
  • FP (False Positive) means that the field was extracted, but does not map to any Gold value.
  • FP, FN (False Positive, False Negative) means that both Gold and Extracted values are present, but differ.
  • FN (False Negative) means that the Gold value is present but does not map to any Extracted field.
• CHILD_NAME: name of a child field.
• CHILD_GOLD: Gold value of a child field. Value of this column depends on the evaluation type.
• CHILD_EXTRACTED: extracted value of a child field. Value of this column depends on the evaluation type.

IePerFieldCSVPrinter

IePerFieldCSVPrinter provides statistics for Information Extraction.

• Input: collection of IeDocumentResult.
• Output: metrics for each field name based on the mapping result between each Gold and Extracted field.

The result is written to the per-field.csv file, as in the following example.

• FIELD: name of a field.
• TP: True Positive. Number of correctly extracted fields.
• TN: True Negative. Always zero for Information Extraction tasks.
• FP: False Positive. Number of extracted fields that are not identical to any gold field.
• FN: False Negative. Number of gold fields that are not identical to any extracted fields.
• P: Precision.
• R: Recall.
• F1: F₁score.
• A: Accuracy.

IePerGroupFieldCSVPrinter

IePerGroupFieldCSVPrinter provides statistics for Information Extraction.

• Input: collection of IeDocumentResult.
• Output: metrics for each group field name based on the mapping result between each Gold and Extracted group field.

The result is written into the per-group-field.csv file.

• FIELD: name of a field.
• TP: True Positive. Number of correctly extracted fields.
• TN: True Negative. Always zero for Information Extraction tasks.
• FP: False Positive. Number of extracted fields that are not identical to any gold field.
• FN: False Negative. Number of gold fields that are not identical to any extracted fields.
• P: Precision.
• R: Recall.
• F1: F₁ score.
• A: Accuracy.

ClassificationGoldVsExtractedCSVPrinter

ClassificationGoldVsExtractedCSVPrinter provides statistics for Classification.

• Input: collection of ClassificationDocumentResult
• Output: mapping result between each Gold and Extracted label for each file.

Results are written to the label-gold-vs-extracted.csv file.

• FILE_NAME: name of a document file from training or evaluation set.
• GOLD_LABEL: Gold value of label for a document.
• EXTRACTED_LABEL: Extracted value of label for a document.
• RESULT: type of mapping between the Gold and Extracted value. It can have the following values:
  • TP (True Positive) means that Gold and Extracted values are identical.
  • FP (False Positive) means that the field was extracted, but does not map to any Gold value.
  • FP, FN (False Positive, False Negative) means that both Gold and Extracted values are present, but differ.
  • FN (False Negative) means that the Gold value is present but does not map to any Extracted field.
• REASON represents information about what caused the RESULT. It can have the following values:
  • EXTRACT_MODEL means that Gold and Extracted values are identical, and were extracted by a model.
  • FAIL_MODEL means that Gold and Extracted values are different, and failed by a model.

ClassificationPerLabelCSVPrinter

ClassificationPerLabelCSVPrinter provides statistics for Classification.

• Input: collection of ClassificationDocumentResult.
• Output: metrics for each label in the mapping result between each gold and extracted labels for each document.

Results are written to the per-label.csv file.

• LABEL: name of a label.
• TP: True Positive. Number of correctly extracted fields.
• TN: True Negative. Always zero for Information Extraction tasks.
• FP: False Positive. Number of extracted fields that are not identical to any gold field.
• FN: False Negative. Number of gold fields that are not identical to any extracted fields.
• P: Precision.
• R: Recall.
• F1: F₁ score.
• A: Accuracy.

Statistics configuration

To configure statistics calculators, define a @Named component in your model configuration which returns a list of StatisticsCalculator.

The following code sample can be applied to an Information Extraction model.

import java.util.ArrayList;
import java.util.List;

import com.workfusion.vds.sdk.api.hypermodel.annotation.ModelConfiguration;
import com.workfusion.vds.sdk.api.hypermodel.annotation.Named;
import com.workfusion.vds.sdk.api.nlp.statistics.StatisticsCalculator;
import com.workfusion.vds.sdk.nlp.component.statistics.IeStatisticsCalculator;
import com.workfusion.vds.sdk.nlp.component.statistics.StatisticsEvaluationType;
import com.workfusion.vds.sdk.nlp.component.statistics.printer.IeGoldVsExtractedCSVPrinter;
import com.workfusion.vds.sdk.nlp.component.statistics.printer.IeGoldVsExtractedPerGroupCSVPrinter;
import com.workfusion.vds.sdk.nlp.component.statistics.printer.IePerFieldCSVPrinter;
import com.workfusion.vds.sdk.nlp.component.statistics.printer.IePerGroupFieldCSVPrinter;

@ModelConfiguration
public class CustomStatisticConfiguration {

    @Named("customStatistics")
    public List<StatisticsCalculator> statisticsCalculators() {
        List<StatisticsCalculator> calculators = new ArrayList<>();
        // add any number of OOTB or custom calculators
        calculators.add(new IeStatisticsCalculator()
                // enable evaluation type (VALUE_BASED is enabled by default)
                .enable(StatisticsEvaluationType.VALUE_BASED).enable(StatisticsEvaluationType.POSITION_BASED)
                .enable(StatisticsEvaluationType.TEXT_BASED)
                // add any number of OOTB or custom printers
                // field printers
                .printer(new IeGoldVsExtractedCSVPrinter())
                .printer(new IePerFieldCSVPrinter())
                // group field printers
                .printer(new IeGoldVsExtractedPerGroupCSVPrinter())
                .printer(new IePerGroupFieldCSVPrinter()));
        return calculators;
    }

}

The next code sample can be applied in a Classification model.

import java.util.ArrayList;
import java.util.List;

import com.workfusion.vds.sdk.api.hypermodel.annotation.ModelConfiguration;
import com.workfusion.vds.sdk.api.hypermodel.annotation.Named;
import com.workfusion.vds.sdk.api.nlp.statistics.StatisticsCalculator;
import com.workfusion.vds.sdk.nlp.component.statistics.ClassificationStatisticsCalculator;
import com.workfusion.vds.sdk.nlp.component.statistics.printer.ClassificationGoldVsExtractedCSVPrinter;
import com.workfusion.vds.sdk.nlp.component.statistics.printer.ClassificationPerLabelCSVPrinter;

@ModelConfiguration
public class CustomStatisticConfiguration {

    @Named("customStatistics")
    public List<StatisticsCalculator> statisticsCalculators() {
        List<StatisticsCalculator> calculators = new ArrayList<>();
        // add any number of calculators OOTB or custom
        calculators.add(new ClassificationStatisticsCalculator()
                // add any number of printers OOTB or custom
                .printer(new ClassificationGoldVsExtractedCSVPrinter())
                .printer(new ClassificationPerLabelCSVPrinter()));
        return calculators;
    }

}