Post-processing examples

The page provides a number of post-processor examples covering a variety of use cases.

note

Normalizers are not standalone components and should be used within a post-processor instance.

Amount normalization

For amount normalization, use the OcrAmountNormalizer class. The normalizer has two main options:

• Fixing OCR errors in numbers
• Formatting numbers and removing currency signs, if any

By default, the #.00 format is used for number formatting, but you can provide your own format. See the following example for details:

OcrAmountNormalizer normalizer = new OcrAmountNormalizer();
normalizer.normalize("1,178.42") => "1178.42";
normalizer.normalize("2,154,60") => "2154.60";
normalizer.normalize("$2,154,60") => "2154.60";
normalizer.normalize("700435.85", "###,###.#") => "700,435.8";

Sometimes, a model extracts fields with a currency sign, or there may be some OCR errors in the amount field.

The following post-processor implementation normalizes $1,234,56 to 1234.56. For more details, refer to the OcrAmountNormalizer class documentation.

public void process(IeDocument document) throws ProcessingException {
    Optional<Field> amount = document.findField("invoice_amount");
    if (amount.isPresent()) {
        Field amountField = amount.get();
        String value = amountField.getValue();
        OcrAmountNormalizer amountNormalizer = new OcrAmountNormalizer();
        amountField.setValue(amountNormalizer.normalize(value));
    }
}

Date normalization

To modify dates to an appropriate format and fix some OCR errors, use the OcrAmountNormalizer class.

By default, the ISO 8601 Date Standard is used: YYYY-MM-DD. You can set your custom date format. See the following examples.

OcrDateNormalizer normalizer = new OcrDateNormalizer();
normalizer.normalize("30 Jun 12") => "2012-06-30";
normalizer.normalize("07/31/2015") => "2015-07-31";

Custom date format:

OcrDateNormalizer normalizer = new OcrDateNormalizer("MM/dd/yyyy");
normalizer.normalize("23-Au0-201 5") => "08/23/2015";
normalizer.normalize("30 ABRIL    \n    >015") => "04/30/2015";

Anchor date for incomplete dates:

OcrDateNormalizer normalizer = new OcrDateNormalizer("MM/dd/yyyy", LocalDate.of(2017, 1, 1));
normalizer.normalize("May 10") => "05/10/2017";

Year and month priority:

OcrDateNormalizer normalizer = new OcrDateNormalizer("MM/dd/yyyy", LocalDate.of(2017, 1, 1));
normalizer.normalize("May 10") => "05/10/2017";
normalizer.setYearPriorDay(true);
normalizer.normalize("May 10") => "05/01/2010";
normalizer.setFirstDayOfMonth(false);
normalizer.normalize("May") => "05/31/2017";

Text-to-date normalization

To normalize text to dates, use the TextToDateNormalizer class.

It normalizes the text representation of dates to formatted dates. You can provide a date format and anchor a date for incomplete dates.

LocalDate anchorDate = LocalDate.of(2011, 12, 13);
TextToDateNormalizer normalizer = new TextToDateNormalizer(anchorDate);
normalizer.normalize("in 3 month") => "2012-03-12";
normalizer.normalize("5 days ago") => "2011-12-08";
normalizer.normalize("after forty one day") => "2012-01-23";
normalizer.normalize("this Friday") => "2011-12-16";
normalizer.normalize("Monday last week") => "2011-12-05";

Custom format:

LocalDate anchorDate = LocalDate.of(2011, 12, 13);
normalizer = new TextToDateNormalizer("dd/MM/yyyy", anchorDate);
normalizer.normalize("in 3 month") => "12/03/2012";

Text-to-number normalization

This normalizer provides the same functionality as the previous one, but for numbers instead of dates. For this purpose, use the TextToIntegerNormalizer class.

TextToIntegerNormalizer normalizer = new TextToIntegerNormalizer();
normalizer.normalize("one thousand two hundred eleven") => "1211";
normalizer.normalize("ninety nine thousand nine hundred ninety nine", "##,###.00") => "99,999.00";
normalizer.normalize("minus one") => -1;

OCR error correction example

After the model results analysis, we see that the zip_code field contains some OCR errors. The following code helps us replace G > 6, B > 8, O > 0.

import java.util.Optional;

import com.workfusion.vds.sdk.api.nlp.model.Field;
import com.workfusion.vds.sdk.api.nlp.model.IeDocument;
import com.workfusion.vds.sdk.api.nlp.processing.ProcessingException;
import com.workfusion.vds.sdk.api.nlp.processing.Processor;

public class ZipPostProcessor implements Processor<IeDocument> {

    public void process(IeDocument document) throws ProcessingException {
        Optional<Field> zipCode = document.findField("zip_code");
        if (zipCode.isPresent()) {
            Field zipCodeField = zipCode.get();
            String value = zipCodeField.getValue();
            String correctedValue = value.toUpperCase()
                .replaceAll("G", "6")
                .replaceAll("B", "8")
                .replaceAll("O", "0");
            zipCodeField.setValue(correctedValue);
        }
    }
}

Field grouping

An AutoML model can extract multiple fields of the same type (for example, Currency, Quantity, Price) that should be joined into groups according to their connection.

AutoML SDK provides two strategies to assign group numbers that come as out-of-the-box post-processors:

• RowBasedGroupingProcessor
• PositionBasedGroupingProcessor

RowBasedGroupingProcessor

The row-based strategy can be applied to documents containing tables:

1. RowBasedGroupingProcessor iterates through all rows in tables and then finds field groups.
2. If a row contains at least one group field, it assigns the same group number to all fields in this row.

In the following example, RowBasedGroupingProcessor will assign three tabnumber group numbers to each row consisting of Account Number, Bank, and SWIFT.

Group Number	Account Number	Bank	SWIFT
0	100500	Private Bank	BOPIPHMM
1	100608	Super Bank	BNORPHMM
2	100609	Super Bank	CHBKPHMM

PositionBasedGroupingProcessor

The position-based strategy can be applied to any document.

PositionBasedGroupingProcessor is based on field indices in a document. It finds group fields in a document and then assigns group numbers according to their indices in the text.

Fields with the same name can't exist in one group. The algorithm adds fields to a current group until the first field with the same name is found. Then, this field is added to a new group (with its number incremented), and all other fields are added to this new group.

Consider the following tagged text sample:

<p><span style="color: rgb(255,102,0);">Samuel Langhorne Clemens</span> (<span style="color: rgb(51,102,255);">November 30, 1835 – April 21, 1910</span>), better known by his pen name <span style="color: rgb(153,204,0);">Mark Twain</span>, was an American writer, humorist, entrepreneur, publisher, and lecturer. Among his novels are The Adventures of Tom Sawyer (1876) and its sequel, the Adventures of Huckleberry Finn (1885), the latter often called &quot;The Great American Novel&quot;. <span style="color: rgb(255,102,0);">Eric Arthur Blair</span> (<span style="color: rgb(51,102,255);">25 June 1903 – 21 January 1950</span>), better known by his pen name <span style="color: rgb(153,204,0);">George Orwell</span>, was an English novelist, essayist, journalist, and critic. He is best known for the allegorical novella Animal Farm (1945) and the dystopian novel Nineteen Eighty-Four (1949).</p>

Fields:

• Name
• Pen Name
• Life Years

The grouping algorithm steps are as follows:

1. Set the current Group Number to zero (0).
2. Go to the first field (tag): Samuel Langhorne Clemens (Name).
   1. Check whether Group 0 already has a Name field → No.
   2. Set Group Number = 0 for this field.
3. Go to the second field: November 30, 1835 – April 21, 1910 (Life Years).
   1. Check whether Group 0 already has a Life Years field → No.
   2. Set Group Number = 0 for this field.
4. Go to the third field: Mark Twain (Pen Name).
   1. Check whether Group 0 already has a Pen Name field → No.
   2. Set Group Number = 0 for this field.
5. Go to the fourth field (tag): Eric Arthur Blair (Name).
   1. Check whether Group 0 already has a Name field → Yes.
   2. Set the current Group Number to one (1).
   3. Set Group Number = 1 for this field.
6. Go to the fifth field (tag) ...

Custom grouping processor

In some cases, you might need to implement a custom grouping strategy. Custom grouping processors are applied to documents with a complex structure, such as:

• Mix of text paragraphs and tables
• Records spread over several table rows
• One table column common for all records

To handle such cases, you can override BaseGroupingProcessor and implement your custom logic in the findGroups() method. You can use the insertGroupIndices() method to set the group index for a field.

For example, you have a document with tables, where one table column (Country) is common for all records.

Country	Account Number	Bank Name	SWIFT
Belgium	100500	Private Bank	BOPIPHMM
Belgium	100608	Super Bank	BNORPHMM
Belgium	100609	Super Bank	CHBKPHMM

In this case, field grouping can be implemented using the following code sample.

public class CustomGroupingProcessor extends BaseGroupingProcessor {

    @Override
    protected void findGroups(IeDocument document, Set<String> groupCodes) {
        Collection<Row> rowAnnotations = document.findAll(Row.class);
        int groupIdx = 0;

        for (Row row : rowAnnotations) {
            List<Field> fields = document.findCovered(Field.class, row);
            List<Field> groupFields = fields.stream()
                    .filter(field -> groupCodes.contains(field.getName()))
                    .collect(Collectors.toList());
            if (!groupFields.isEmpty()) {
                insertGroupIndices(groupFields, groupIdx);
                groupIdx++;
            }
        }

        //country belongs to all groups
        Field countryField = document.findField("country");
        insertGroupIndices(countryField, IntStream.rangeClosed(0, groupIdx).toArray());
    }
}

Post-processing for classification models

Below you can find an example of processing for a classification model where the Document label is changed from Deutsche bank to DB and then set score to 1.

public void process(ClassificationDocument document) throws ProcessingException {
    Label bank = document.findLabels().stream()
            .findFirst()
            .orElse(null);
    String name = bank.getName();
    if (bank != null && "Deutsche bank".contains(name)) {
        document.remove(bank);
        document.add(Label.descriptor().setName("DB").setScore(BigDecimal.ONE));
    }
}