Edit

Study documents

What are the basics?

The first necessary information SME should share with DA includes:

1. What layouts we can distinguish in the documents' flow
2. What layouts are outliers in the flow (fewer than 30 samples)
3. What are the formats of documents (pdf, picture, spreadsheet, etc.)
4. Any exception cases
5. What output format of the data is required, if there is a special one, etc.
6. If there any mapping which should be applied

important

These are the first pieces of information necessary to make the entire process less problematic.

Now, let's learn what we need to start the process. The following content reflects Step 1 of the workflow, "Data Analyst studies the requirements."

Study documents' logic

When we prepare documents to be a part of an automation process, several things matter. These are:

• business rules of the documents
• aptitude of data in the document for machine learning

Usually, all these refer to selection of where to tag fields in the document.

Let's consider two examples:

1. In the first sample, Business ID occurs on all pages in the document, and the value doesn't change. So from a business perspective, it doesn't matter where this value is tagged. Only one place should be chosen, on the Data Analyst's advice. Usually, it's the on the first page or a place where OCR does not corrupt the value.
2. The other case is with amounts, which can't be tagged anywhere in the document because there may be subtotal, total, and balance due, and business logic dictates which one we need to extract or whether these should be three separate fields.

The main point here is that the Data Analyst should know if there are cases where a particular field should be taken only from a specific place. In all the rest of the cases, the DA should advise where is best to tag.

important

Documents' format and structure should remain the same; otherwise, it will be not possible to expect the same extraction results. So if in a data set, all invoices contain only one page, in production only one-page invoices should be expected. If invoices are scanned together with the proforma invoice, they should be scanned the same way in production.

Any changes in the original document or document layout requires model retraining.

Check documents' quality

Original documents' quality influences model results a lot. When possible, documents with the highest OCR quality should be selected for the data set collection. When selecting documents, those with shadows, stains, any lines or dots from scanning should be excluded. Any torn or crumpled documents also should be excluded. When documents are scanned for data set collection, they should be scanned with a high resolution and correct position on the page. Otherwise, extra time will have to be spent developing OCR documents to make the most of them in terms of character recognition and model extraction. It's necessary to remember that if the value provided in the PDF is lost during OCR, it can't be extracted by a model.

It's also critical to ensure the same parameters of scanning are kept in production so that OCR produces the same output and the model works on the same HTML structure of the document as it learned during training.

Resolution

Position on the page

Crumpled sheets, stains, etc.

Noise

A high-quality scan is one that is easy for everyone to read. High-quality scans should be free from:

• cut-off text
• crooked pages
• dark gutters (the margins where shadows occur from curvature in the book's spine)
• poor contrast
• pages that are rotated 90 or 180 degrees
• handwriting
• highlighting
• underlining
• watermarks/coffee stains
• blur

caution

Original documents' quality/scanning quality sets expectations for documents' structure and HTML structure after OCR, and thus should not change much. Otherwise, model retraining will be required for success.

Make sure distribution is right

The main rule of the data set is that it should be well-distributed, which means the distribution of documents in production should be followed in the data set. This implies that representation of different layouts (documents of different vendors) should be distributed in the same proportion within the data set. There are two aspects of distribution to remember:

• A data set should have as many various samples of documents as possible so that a model can work well on the whole document flow in production.
• A data set should contain enough samples of rare documents and rare fields.

To sum up, it won't work to put into the data set a range of 100 documents from 80-90 different vendors, with 1-2 documents from each vendor. What works well for effective model training is stability, not variety. So there should be sufficient quantity of documents for each layout (at least 30, and more for better model results, depending on the quality of documents).

Some data is present only in particular documents or in particular cases. So these fields are called "rare" (if present in less than 30% of the documents). For effective model training, it's critical to increase the number of samples containing these fields so that the model can learn well, produce stable features, and process such fields in production.

note

It's better to cover as many layouts as possible during training. From a best practices perspective, it's more effective to concentrate on the top 20 or 30 vendors, who might make up 90% of the workflow, and consider including the other 10% vendors in the second round.

Consider grouping documents

Splitting into batches is often used to smooth the process of data set preparation. The batch of documents is a set of typical documents gathered by some criteria. The criterion is often the same layout; most commonly, it's from one vendor. Splitting aims to increase tagging efficiency and speed, and contribute to data set quality because it's proven that tagging of similar documents all at a time is more effective than tagging of various documents.

SME should tell DA what criteria or keywords can be applied while splitting documents into batches. If it's possible to use historical data, it's highly recommended to split documents before the review by the DA. That can save up to a week in the skill development. The most typical criteria for splitting into batches are: type of document, vendor and language. When necessary, other criteria can be applied, as the number of documents per layout should be substantial (for example, 50 or more). The Data Analyst, if there is a need, may ask to increase the number of documents.

Move further

In the next part, we will learn how to prepare data sets collected for tagging.