Define labeling logic

Goal of the step: Define correct labeling patterns for each field in each layout:

• Value shape
• Value boundaries
• Position in the document
• Context
• Acceptable and unacceptable cases due to structural changes after OCR

Output of the step: Precise unambiguous instructions for every field in each batch.

To guarantee high quality of a dataset and reduce labeling mistakes, it's necessary to define a precise and unambiguous logic for each field in each batch in advance so that the SME can clearly understand where and how to label a field.

Steps

Identification of such logic requires a thorough step-by-step analysis of documents from each batch:

1. Take about 20 samples from each batch.
2. For each field, define where and how to label it. Take into account the factors described below.
3. Check that the defined rules correspond to labeling requirements.
4. Describe the logic with rules and screenshots in instructions.

Factors

None of the factors listed below should be ignored as all of them together help identify the best place for each field to be labeled.

Number of field occurrences

Field occurrence means the number of times the same value of one field appears in a document. Usually, the number of occurrences is considered for single-value fields. See the following example.

The secretary_of_state field occurs in the document only once, so there will be no confusion in labeling this field and no specific instructions are required.

Address information (street, city, zip, and so on) occurs in two different blocks, so the instructions should specify only one place to be used for this field.

Document structure

After OCR, the information in an XML document can be structured in the following ways:

• As a free text inside of one or several strings
• As a table with different number of cells
• With values divided into several parts

Examples of different structures:

Value and surrounding context

For choosing specific occurrences to be labeled, WorkFusion recommends considering the value itself and the surrounding context.

note

OCR quality is very important for choosing the specific occurrence of a field for labeling.

OCR quality

Both the value and surrounding context should be the most stable after OCR, which means:

• Always recognized
• Not corrupted
• No mistakes at all or only minor mistakes that are systematic and can be fixed by applying a common rule
• Has the same structure

Content

Identify the value unambigiously:

• WorkFusion always means "WorkFusion".
• Workfusion always means "WorkFusion".

But WF might not always mean "WorkFusion".

• WF sometimes means "World Freight Group", sometimes "WorkFusion".
• CC sometimes means "Coca-Cola", sometimes means "Chemours Company".
• UBS can be interpreted as:
  • "UBS acting through its London branch"
  • "UBS Russia"
  • "UBS Group AG"

Content should be informative, uniform, unambiguous across all documents.

It is also better when there are some key words helping to identify the correct value, like:

• "number"
• "ID"
• "name"
• "client"

Example of labeling logic development

We are not going to consider 20 documents now. Let’s take a look at just four to define specific occurrences for labeling the business_id field in a batch (New Hampshire).

Document A

• Original

  

• OCRed

  

Document B

• Original

  

• OCRed

  

Document C

• Original

  

• OCRed

  

Document D

• Original

  

• OCRed

  

We have two occurrences for business_id.

Content factors have the same characteristics in both cases:

• The value is unambiguous in both cases, that is, it always means specific business_id, and it cannot be interpreted in any other way.
• The content of the surrounding context is good in both cases. For every occurrence, it goes after the "Business ID" phrase.

To analyze the OCR quality factor quickly, let’s create a table:

Document	OCR quality for 1st occurrence	OCR quality for 2st occurrence
A	+	(table)
B	+	(not a table)
C	+	- (not recognized)
D	+	- (context structure is different)

So, in this case, it’s clear that it is better to choose the first occurrence as the latter is more stable from the OCR quality point of view, while the other characteristics are the same.

As a result, the most convenient way to deliver this information is to provide a screenshot to the user, like this: