Prepare data set
Training Set
Training set is a set of documents (invoices, emails, contracts, etc.) used for training an ML model. For Information Extraction, a training set consists of documents which contain gold values (values surrounded by special tags) for a set of fields.
Fields — attributes that should be extracted from the documents (for example, date, email, company name, company address).
Each tagged value is an object for some particular field. It is surrounded by a special tag with special attributes. This text and the tag are called a gold tag or a gold value.
Here's an example of extraction tag appearance on UI.
And here's the extraction tag appearance in HTML structure.
<span class="font5">
"PGS 1 $13.00 DF $"
<total_transfer_tax data-toggle="tooltip" style="background-color: rgb(183, 151, 98);"
class="extraction-tag" tagorder="1" appendorder="2" data-value="24,20"
data-original -title=" Total Transfer Tax (99999.99)"
confidence="1/1">24, 20</total_transfer_tax> ==
</span>
Field name is used as a tag name and its attributes contain the mark that it is a special tag class="extraction-tag" the text in which is a gold value, as well as some information about this value. The most important attributes of a tag are the following:
data-value— a gold tag attribute that will be returned as extraction result (what a user will see, if they try to save extraction results as JSON or CSV). May be changed in a manual task or by post-processing as final result does not necessarily equal to the tagged string. Note that ML will be focused on tagged string, not data-value. On the picture above you seedata-value="24.20". Below is the case when thedata-valueis not equal to the tagged text.tabnumber— attribute showing that the tag belong to some group of tags and the number of this group. We will give more details about this attribute in Tables Processing.appendorder— attribute defining in which order different chunks of the same tag should be appended to each other.
Data Set Requirements
The first requirement for data set concerns its size. It should consist of at least 500-1000 documents for unstructured data. For highly structured data the training set can be smaller - for example, if your population contains only one layout and all documents have same structure, 100 of them would be enough for training.
A data set for machine learning should be tagged in a particular way only - the model does not treat text same way as people do. While people can perceive text (be it names, separate words, numbers, etc.) in its entirety, understanding the meaning of what they see and being able to identify why it makes sense, to the model any text is just strings of symbols organized in an html or xml tree. Its goal is to identify a value and the context surrounding the value.
Context is basically everything that surrounds the tagged piece of text, its position on the page and in some cases its position relative to other important pieces of information. Context is the most important source of features and is used to determine where a gold value should be searched.
On this picture "R" is tagged as "currency" and 316,920.00 as total_amount. All the rest of the text is a short context for these two values.
Gold values in the data set are the objects that should be classified as "the value that should be extracted". The set of features is defined on the whole data set (all the documents). In each document for each gold value feature values will be calculated and weighed during the training process and the model (the function) will be built. When the model is trained and you run it on some data, it will extract the strings that can be classified as "the value should be extracted" by their features (they are situated in the part of the feature space that contains objects that should be extracted).
Values themselves are the source of features as well. For example, CUSIP number that is used for classifying financial instruments worldwide has a definite shape and check sum, so any extracted value for CUSIP can be validated by these criteria. Suppose that our documents contain more than one CUSIP and we are interested in some particular one - for example, facility CUSIP. In this case CUSIP shape and check sum can be used for identifying all CUSIPs in the document while their context can be used for identifying the specific CUSIP we are interested in.
The data set quality is very important for machine learning as wrong values or wrong context are the source of incorrect features and weights, which affects the model's quality. "Garbage in - garbage out" is the main principle of supervised machine learning, which means if the training set is not good enough and big enough, the model won't show good results. And the data set quality is one of the most important DA responsibilities. A good data set meets the following requirements:
Diversity
Objects from bad-represented or not represented layouts may have feature values which make the model treat them like outliers. In best case they will be ignored, in worse case they will generate additional false positives. That's why it's important to have enough representation of all layouts and features we aim to extract, as in the reality.
Consistency
Consistency is one of the most important requirements to your data set. Each field should be consistently tagged across all the documents (or a certain layout): for example, if an invoice has the same "po_number" occurring on all the pages of the document, we should always label the one at the same location (on the same one page across all the data set; for example, on the first page). Otherwise, it will send a confused signal to Machine Learning algorithm:
If a layout contains the same value several times in different parts of a document (it's rather common case for some fields, e.g total invoice amount, taxes), it's recommended to chose one context where it will be tagged in advance and stick to this context across all the documents of this layout. If we have the following situation for some field:
| Context/Value | Value 1 | Value 2 | Value 3 | Value 4 | Value 5 |
|---|---|---|---|---|---|
| Context 1 | tagged | tagged | |||
| Context 2 | tagged | ||||
| Context 3 | tagged | tagged |
The model may treat it in the following way:
| Context/Value | Value 1 | Value 2 | Value 3 | Value 4 | Value 5 | Conclusion |
|---|---|---|---|---|---|---|
| Context 1 | Correct | Incorrect | Incorrect | Incorrect | Correct | Not extracted in 60% => most probably should not be extracted => "not to extract" => FN |
| Context 2 | Incorrect | Incorrect | Correct | Incorrect | Incorrect | Not extracted in 80% => most probably should not be extracted => "not to extract" => FN |
| Context 3 | Incorrect | Correct | Incorrect | Correct | Incorrect | Not extracted in 60% => most probably should not be extracted => "not to extract" => FN |
Here the fact that same value was tagged in different context across documents resulted in the model deciding that the value should not be extracted at all.
As we mentioned, context and a document structure around the tagged value is an important source of features. If the value is not extracted in the context the model expects it to be, it may learn that the values in this context may be incorrect and should not be extracted at all. It will generate additional false negatives in extraction results and affect Recall.
Another bad effect may take place if the model defined the features for context in not an optimal way. For example, in one type invoice_date is given in the document after "Invoice date:" and after "Date:" and is tagged in both contexts. Documents also contain "Payment date: {date in the same format}". In this case the model may find "the context like required" and extract payment date instead of invoice date as has incorrect feature "date: {value that should be extracted}". It will generate additional false positives in extraction results and affect Precision.
If a Data Scientist requires to tag all the appearances of the field (in case they want to let the model learn all the possible contexts), a field should be tagged in each of its appearances (missing any of them is a gross error). However, it's a rare case as long as we understand that a human factor may result in missed and untagged appearances and, futhermore, in additional false values, hence, it's a very rare practice.
Completion
Completion has two different aspects:
- If a correct field is given in the document, it must be tagged. Otherwise:
- The model may treat it as incorrect and generate additional FN by the same logic as in the previous example (affects Recall).
- If its feature values will be enough to be extracted and it's extracted, when counting evaluation statistics we will have a FP here because the gold is empty (affects Precision).
- If a field contains multiple words (in ML space, we call it a "token"), we should always label the entire string (with all the words) belonging to the field.
For example, we should extract supplier_name and it's always a company name. We see "WorkFusion Inc." and think that "WorkFusion" is important information and should be tagged and "Inc." is not very important and we don't tag it. The model gets the information that only a part of a string that contains supplier_name should be extracted as supplier name. And when the model "sees" "The Bank of America", it will extract "The Bank of" because previously we "told" it that a part of company name is correct value and entire string is incorrect value.
| Gold data | Extraction results |
|---|---|
WorkFusion Inc. | The Bank of America |
In case you need to exclude some part of the string from the values, for example there is a requirement that supplier_name shouldn't contain legal endings, it's recommended to tag the full value, correct data-value and apply post processing (in case of legal endings, it's better to use dictionary-based approach) to the model results, so that you have full control of what is changed.
Normalization
If the value can be given in many formats, it should be normalized to the same one format across all the gold data.
For example, date can be given as 01/18/2017, January 18, 2017, Jan-18-2017, etc. All these option are correct but presented in different format - so we need to normalize them to a common format by selecting a mask for date in the manual task designer.
If the values should be mapped and some reference data is planned to be used, the reference data should be added as a dropdown in a manual task.
If a value should be normalized in some special way (for example legal endings should be removed from a company name) the data-value should be corrected in a manual task (or later in post processing).
If the values are not normalized, additional false positives may appear in extraction statistics (which affects Precision).
| Gold Value | Extracted Value | Decision | Comment |
|---|---|---|---|
| Jan-18-2017 | 01/18/2017 | FP | Cannot be compared with the extracted result correctly |
| 18.01.2017 | 01/18/2017 | FP | The date wasn't normalized in gold data and cannot be compared with the exacted result correctly |
| WorkFusion Inc. | WorkFusion | FP | The extracted value is correct and normalized but the gold value is not normalized, therefore, they cannot be compared correctly. |
| WorkFusion Inc. | WorkFusion (UK) Limited | FP | The extracted value was derived from the company name and its address (which is UK address) using reference data, but it wasn't done in gold value, therefore, gold and extracted values cannot be compared correctly. |
We covered the most common requirements to a data set. A few other requirements for a training set include the following: make sure that there are no conflicting values in different fields (for example, “Male” and “Pregnant”), number values are not out of range (“Age=1000”), missing values (either empty fields, or the fields that are recognized as containing backspaces but which are actually empty) etc.
Typical Cases of Incorrect Tagging
Extra Data Tagged
Tag E-mail contains phrase "E-mail" which doesn’t relate to the tag.
Incomplete Data Tagged
Tag Invoice number doesn’t contain full information from the required field.
Optical Character Recognition (OCR)
In 80% of use cases WorkFusion AutoML processes documents that are scans, unsearchable PDF files or images. All WorkFusion models require HTML, XML or plain text documents as an input. Therefore, to get the documents in suitable format, we use 3rd party OCR (Optical Character Recognition) software in order to convert images and PDF documents into text (plain text, XML, or HTML).
To achieve that, WorkFusion uses ABBYY OCR which offers an out-of-the-box solution: we don’t need to write or maintain any custom code. However, each iteration is chargeable: we pay for each page.
Before sending documents for manual tagging and preparing the data set, OCR quality should be checked, because OCRed documents structure is an important source of features. The original document structure is not always exactly reproduced by OCR. There can be some difference between original and OCRed document. For example, different number of lines in text, different tables formatting, additional new lines and paragraphs. So, OCRed document looks not exactly the same as original document, but it doesn't prevent ML to process it and handle these changes correctly, especially if they are reproduced across all or most of the documents within one layout.
OCR can make mistakes in recognition of some characters. For example, zeros ("0") can be recognized as a letter "O". Such mistakes can be handled by adding post-processing logic that replaces incorrect symbols or symbol sequences with correct ones. If OCR mistakes in values or changes in document structure can be fixed by post-processing, they don't require changing OCR settings.
In some cases OCR results are not accurate enough (hand-written text on documents, images with low DPI, twisted scans, crumpled paper, scans of photocopies, etc.) which causes certain problems (table margins, wrong or missing characters etc., appearance of additional structures that corrupt values in the way that ML cannot find them and extract correctly).
Therefore, OCR results may be not accurate. Below are some common problems that you may see after OCR:
Problem |
Solution |
Data Set Compliance |
|---|---|---|
| Text is missed. | Change OCR parameters. |
Exclude |
| Characters are wrong, but can be recognized by human. | If it's an occasional mistake, that appears in some documents of a layout, it should be corrected manually in HT when creating the gold set, and handled by post-processing in automation results. If some value is corrupted in the whole layout and cannot be corrected by post-processing (only removed), OCR should be improved. There can be some exclusions. E.g.: company name is given only in logo and it cannot be recognized well by OCR. |
Include in case of occasional mistake. |
| The value is totally corrupted and cannot be recognized by human. | If the value itself and it's context is totally corrupted and without checking the original document you cannot understand what should be in this part of document, it should not be tagged. The document should be excluded from the training set, because it won't let the model learn anything useful and may be the source of wrong features and weights. If such documents are rather regular cases in the common documents flow and are not caused by the original documents quality, it's recommended to improve OCR quality. | Exclude |
| Corrupted, irregular structure. | Text may be converted into a tables with random structure that won't be reproduced in other documents. If the value to be extracted is in the table, it's recommended to exclude such documents from the training set as they may be a source of wrong features and weights. If such documents are rather regular cases in the common documents flow and are not caused by the original documents quality, it's recommended to improve OCR quality. | Exclude |
It's recommended to exclude the document's with strongly corrupted structure, missed text and values that cannot be recognized by human from the training set, but they should be included into the test set (in ideal case in the same proportion as they are represented in regular documents flow). If they are excluded from ML results evaluation, the statistics will be overestimated and won't be reproduced on the customer testing phase.
Remember, a good enough and big enough training set is an important factor in ML success. "Garbage in – garbage out" is how ML works. If OCR quality is low, wrong features will be defined and the model quality will be low. If the training set contains mistakes, they will be reproduced in evaluation and you may not achieve your targets (each use case has targets for Precision and Recall). If gold data precision is lower than the target for the use case (for example, your target is P=95% and gold data contains 15% errors), you will never be able to achieve such target. So, be mindful about the training set quality.