Results analysis

Goal: To analyze the results of the machine learning (ML) model and propose ways to improve them.

Input: Test set, trained model.

Output: Report with statistics on test extraction, proposals on ML model improvements.

Results analysis steps

The results analysis stage includes the following steps:

1. Run ML model training.
2. Run Information Extraction (IE) or Classification on a test set.
3. Get ML model results.
4. Run a Business Process (BP) to get detailed statistics.
5. Analyze results to find the reasons for mistakes and provide solutions.

Run model training and IE or Classification on test set

After splitting the dataset into training and test sets, the Data Analyst (DA) and ML Engineer launch model training and then run IE or Classification on a test set.

Make sure that the input file provided for extraction includes the following information:

• Column with links to the original documents
• Column with links to the OCRed documents
• Column with labeled text (as XML content)

This is important because this information will be used in the next statistics calculation step.

Get model results

After the model is executed on the test set, we'll get a file containing all the information from the input file and a column with extracted text in the XML format:

• original_document_link: a link to the original document in the file storage
• ocr_document_link: a link to the OCRed document in the file storage
• tagged_text: gold data content
• tagged_text_tagged: extracted values

Run Business Process to get detailed statistics

To get detailed information as well as accurate statistics per field, you can use the Statistics aggregation v2.0 Business Process. This Business Process transforms the results of the model instruction into a file that is structured and prepared for analysis. The file also contains detailed extraction results on each document per field and aggregated statistics per field.

The final result will look like this:

Raw data

Deliverables

Running statistics aggregation BP

Input for the BP: a CSV file with IE results

Output: an Excel file with calculated per field statistics

To launch this BP, proceed with these steps:

1. Upload the IE results as input data.
2. On the Design tab, find the Params block .
3. Specify the following parameters:
   • datastore_name: create a Data Store manually in advance; otherwise, it will be created automatically.
   • gold: name of the column with manually labeled data.
   • extracted: name of the column with extracted data.
   • documents_count: number of records in the input file.
   • S3_bucket/folder/name_to_save: specify a bucket, folder, and file name where results are to be stored in S3.
4. Run the Business Process.
5. Go to the Results tab and download Statistics.xlsx.

Analyze results: find mistakes and provide solutions

At this stage, the Data Analyst should thoroughly analyze the model results to understand possible reasons for each mistake type. In case of Information Extraction, there are five possible result types:

• TP (True Positive)
• TN (True Negative)
• FP (False Positive)
• FN (False Negative)
• FP-FN (False Positive - False Negative)

Type of value	Definition	Gold value	Extracted value
TP (correct)	The value should be extracted, and it is extracted by the model.	WorkFusion 10 000.00	WorkFusion 10 000.00
TN (correct)	The value should not be extracted, and nothing has been extracted by the model.	-	-
FP (mistake)	No value should be extracted, but the model has extracted something.	-	WorkFusion
FN (mistake)	The value should be extracted, but the model has extracted nothing.	WorkFusion	-
FP-FN (mistake)	The value should be extracted, but the model has extracted it with a mistake.	WorkFusion Exxon Mobil 10000.00 01/01/2018	W0rkfusi0n Exxon 100000 1st of January, 2018
	The value should be extracted, but the model has extracted something different.	WorkFusion 10000.00 01/01/2018	Company name: 1700 08/10/2017

For more details about the metrics and how to calculate them, go Report | Metrics.

Types of mistakes, reasons, and solutions

As TP and TN values are absolutely correct model answers, pay attention to FP, FN, and FP/FN ones.

False positives (FP)

FP mistakes are unacceptable at this stage. The DA should prepare a dataset attentively in accordance with defined labeling rules and instructions.

Incorrect grouping for multi-value fields

Solution: implement correct grouping in post-processing.

Missing values in test set

Solution: correct gold data, label or re-label all the existing values that were missed or wrongly labeled, or exclude such records from the test set.

Mistakes in test set, totally incorrect values are labeled

False positive, false negative (FP/FN)

Insufficient normalization (extra symbols, different data types)

Solution: normalize values in post-processing.

Inconsistent gold data

Variations in values: for example, the invoice_number field has xxxxx as the gold value and xxxxx HAB as the extracted value, and both are correct from the business point of view but are not equal and cannot be compared to each other.

Solution:

1. Correct all inconsistencies in the gold data:
   • Pre-process or normalize gold values.
   • Re-label or exclude incorrect documents.
2. If previous steps were applied, re-train the model.
3. If gold data wasn't corrected, try to normalize values in post-processing.

Incomplete labeling in training set

Solution: same as above.

OCR errors in extracted values

Example

Solution:

1. Analyze whether there is any logic that allows correcting these mistakes without generating other mistakes on the entire dataset and possible unseen data. If yes, implement corresponding post-processing.
2. If a rule can cover only part of cases without creating extra FPs, try to define the remaining part to remove these values so that they are handled manually.

Specific or broken document structure

OCR makes it impossible to label the value completely.

Solution:

1. Check whether any logic can be applied to extract the value completely in post-processing without creating additional FPs. If yes, such post-processing should be applied.
2. If post-processing cannot help, additional components should be added to the model with AutoML SDK.

False negative (FN)

Not enough examples in training set

• Small overall number of some sample fields in a training set
• Small number of sample fields in specific document structures

Should be identified and communicated in advance.

Solution: increase the number of examples in the training set and (or) retrain the model. Additional components can be added using AutoML SDK.

Labeling inconsistency in training set

The field is labeled in different positions.

Solution: correct or exclude inconsistencies for the field in a training set and retrain the model.

This kind of mistake shows that dataset wasn’t prepared properly, which is DA's responsibility.

New document structure after OCR within known layout

In a test set, a document of a known layout that differs completely from the majority of other documents of the same layout due to different initial quality after OCR. That is, the new layout is badly represented in the training test and consequently is badly recognized in the test set.

Solution:

• If the case is valid, escalate in advance that it is badly represented.
• If the case is invalid, estimate the impact of such documents and escalate to the Delivery Manager.

It is important to have enough documents of some specific structures where this field is represented in a way that's not very simple for extraction.

Complicated dependencies in extracting logic

Dependencies that haven’t been found by the out-of-the-box model.

Solution: use additional components from AutoML SDK.

Possible reasons for mistakes

Mistakes in gold data

If a dataset is collected thoroughly in compliance with defined labeling rules and instructions, this reason is invalid.

Long free text

For example, it was decided to label and, consequently, extract address as one element:

In such a case, the model considers an address line just as some free text without any specific shape and format. Moreover, there is high possibility that different parts of the text line will be located in several different XML elements.

Solution: check if this value can be split into independent parts. For example, an entire address line can be very difficult to extract as address formats usually differ from country to country or even in different document templates. So, the model will be more confident about extracting the city, street, and zip code separately.

Be careful with value splitting. Values like company_name cannot be split as their parts cannot be considered independent and make sense only as a whole name.

Unusual or broken document structure

When surrounding values were moved towards another value.

Solution: try to define some rule common for all such records that makes it possible to extract these values correctly. If such a rule is found and is applicable for post-processing, use it.

One more option is to collect all such cases together and use a separate model trained on such particular cases. This option should be communicated to the Delivery Manager.

During the analysis of the ML model results, the Data Analyst should always consider every mistake in terms of its frequency and impact on the business.

Frequency

Define whether a mistake is a single case or a regular one.

• Exception case: it can be easily removed from the dataset without serious impact on statistics. Special attention should be paid to regular and frequent cases.
• Regular: frequent typical mistake, the correction of which can significantly improve the statistics (different spaces, commas, dots, and so on) without affecting the value itself.

Business impact

Define how critical the mistake is to define its priority for post-processing or extended ML implementation:

• Mistakes in critical fields that can affect business.

For example, we need to extract the price of product items our customer will deliver to the requester. The price provided in the document is $.5525, which means 0.5525 USD, but the extracted price is 5525, which is 10,000 times bigger than the true value.

Be very careful with such fields. Even if one symbol is replaced or damaged (for example, during OCR) and therefore a field value cannot be defined with 100% confidence, it's better to remove the extracted value completely and handle it manually.