9.3

Edit

Post-processing acceleration

Post-processing utilizes normalizers. Before version 9.3, AutoML included a number of out-of-the-box normalizers, but the final post-processing configuration was written by Machine Learning Engineer. MLE had to choose the combination of normalizers himself for each case.

Normalizer Search Engine, introduced in the 9.3 release, tests different combinations of normalizers for each field to select the optimal chain. The final choice is the most useful and the safest for particular case. This minimizes the practice of designing custom post-processors, so MLE doesn't have to write a post-processor in the majority of the cases.

Normalizer Search Engine works especially well on numeric fields: numbers, dates, amount, and price. Quality of these fields can improve up to 70% in comparison with the same IE flow without the post-processing. Normalizer Search Engine is also useful for select one, e-mail, currency, address, text fields, and so on.

Featured normalizers

Normalizer type Description Example: before normalization Example: after normalization
Text normalization for all text fields UpperCaseNormalizer Modifies text to contain only upper-case letters for the specified locale. 
aBCdE
 
ABCDE
LowerCaseNormalizer Modifies text to contain only lower-case letters for the specified locale. 
aBCdE
 
abcde
CapitalizeNormalizer Capitalizes all the whitespace separated words in a string. Only the first letter of each word is changed. 
aBC bcC cCD
 
ABC BcC CCD
TrimmerNormalizer Removes spaces at the beginning and the end of the string. 
" aa a    "
 
"aa a"
WhitespaceNormalizer Replaces multiple occurrences of any whitespace symbol with one standard whitespace symbol. 
"cat     dog"
 "cat dog"
WhitespaceRemoveNormalizer Replaces all occurrences of any whitespace symbol. 
"cat     dog     "
 "catdog"
PunctuationRemoveNormalizer Removes all occurances of punctuation marks. cat,dog catdog

MapDocumentTextToValueNormalizer

Text normalization for select one/optional one field types based on the automatic dictionary provided in resources and the nearest neighbors strategy. Substitutes the answer for the value from the dictionary, if the similarity score between them is acceptable.

Formatting DateFormatNormalizer Date format normalization for date fields based on automatic date-month order determination for given locale. If date format pattern is not provided, the default one ("MM/dd/yyyy") will be used. 1989,10,22 10/22/1989
AmountFormatNormalizer Amount formatting normalization for amount fields based on automatic locale determination used for delimeter choice. If amount format pattern is not provided, the default one (" #.00") will be used. 4,32 4.32
DecimalNumberFormatNormalizer Number formatting normalization for number fields based on automatic locale determination (used for delimeter choice). 3,14159265359 3.14159265359
OCR error correction OcrDateFixNormalizer Removes OCR errors from the date string. Ju1 20, 2015 Jul 20, 2015
DictionaryCorrectionNormalizer Removes OCR errors from the string based on the dictionary automatically gathered by GoldDataAnalyzers. l 5QO 1 500

UsStatesAbbreviationNormalizer		 Replaces the name of the US state by its abbreviation for address fields. Oklahoma OK

For the Data, Amount, and Number fields, the formating pattern should be provided in the field information normalizationPattern attributte, otherwise the default patterns are used: MM/dd/yyyy for the Date fields, #.00 for the Amount fields

Date formatting configuration:

new FieldInfo.Builder("invoice_date")
        .type(FieldType.INVOICE_DATE)
        .multiValue(false)
        .property(GenericPipelineConfiguration.PARAM_NORMALIZATION_PATTERN, "MM/dd/yyyy")
        .build()

Configuration

The new components include:

  • GoldDataAnalyzersConfiguration: a set of analyzers grouped by the field type. Gold Data Analyzer finds patterns and similarities in training data and stores them in resources.
  • GenericNormalizersConfiguration: a set of normalizers grouped by the field type. Normalizers Search Engine uses it to find the best chain of normalizers. Normalizers use the output of gold data analyzers.
  • BestNormalizersProcessor: reads the serialized best normalizers chain and applies them to the corresponding fields.

Generic IE configuration

@ModelConfiguration()
@Import(configurations = {
        @Import.Configuration(CandidateFeatureExtractorsConfiguration.class),
        @Import.Configuration(GenericIeAnnotatorConfiguration.class),
        @Import.Configuration(GenericNormalizersConfiguration.class),
        @Import.Configuration(GoldDataAnalyzersConfiguration.class),
        @Import.Configuration(GenericHpoConfiguration.class),
        @Import.Configuration(GenericIePipelineConfiguration.class),
        @Import.Configuration(GenericIeMlConfiguration.class),
        @Import.Configuration(GenericIeParserConfiguration.class)
})
public class GenericIeHypermodelConfiguration {

    @Named("basePostProcessors")
    public List<Processor> getPostProcessors(DefaultConfigurationContext configurationContext) {
        return Lists.newArrayList(new BestNormalizersProcessor(configurationContext.getResource("/lookup/normalizers")),
                new CheckerModelPostProcessor(),
                new DataValueNormalizationProcessor(),
                new RowBasedGroupingProcessor());
    }


}

Execution flow

  1. Gold Data Analyzer specified for the selected field is executed. The output files (local.json, ocr-corrections.csv) are stored in the resources directory.
  2. All possible combinations and permutations of fields normalizers are generated.
  3. Statistics is calculated and stored for all generated combinations in the normalizer-statistics.csv file.
  4. Best normalizers are serialized in the best-normalizers.json file.
  5. The best-normalizers.json file is used by BestNormalizersProcessor for post-processing.

The structure of files is as follows:

output
    └──resources
        └──lookup
            └──normalizers
                ├──email
                │   └──normalizer-statistics.csv
                ├──invoice_date
                │   ├──best-normalizers.json
                │   ├──date-format-priority.txt
                │   └──normalizer-statistics.csv
                ├──invoice_number
                │   ├──best-normalizers.json
                │   ├──normalizer-statistics.csv
                │   └──ocr-corrections.csv
                ├──price
                │   ├──best-normalizers.json
                │   ├──locale.json
                │   └──normalizer-statistics.csv
                ├──product
                │   └──normalizer-statistics.csv
                ├──quantity
                │   ├──best-normalizers.json
                │   ├──locale.json
                │   └──normalizer-statistics.csv
                └── total_amount
                    ├──best-normalizers.json
                    ├──locale.json
                    └──normalizer-statistics.csv
← PreviousNext →