10.1.6.2

Edit

Labeling and validation

Goal: Get tagged data set of high quality

Input: Manual Task, batches of documents

Output: Reviewed data categorized by groups based on the type of inconsistency found.

When there are at least some batches tagged, a Data Analyst can proceed with data validation. There are two common ways to validate data.

  1. Validate in browser mode.
  2. Validate via creating a new Manual Task.

Let's see each approach in detail.

Validating in browser mode

As an output of labeling stage, the Data Analyst gets a CSV in which tagged text is presented as XML content. To use this method of validation, it's necessary first to convert tagged XML content into links to HTML files. For this purpose, a business process — "Save tagged content to s3" — can be used.

Generating links

Input: A snapshot from a Manual task with a column with tagged text (XML, HTML). The column name should be tagged_text.

Output: CSV with column doc_page_xml_link that contains links to tagged documents on S3.

Download a business process to import to Control Tower.

As a preliminary step to validation by opening links, it's possible to look through data values.

In an Excel file, create a filter for all the required fields and check whether there are empty cells or suspicious values that cannot be the value of this particular field. We can check it because the format and expected content for every field is known, so if anything is incorrect, it is easy to identify and mark them as incorrect without even opening the tagged document itself. Most common examples:

  • The name of the field is company_name but the data value is California, which is the characteristic of state and not the name of the company.
  • Some odd information was tagged: for example, the field company_name data value is “Corporate Name: Fitness-And-You, Inc.”.
  • required fields include blank cells (no data values); inspect separately to define the reason for information missing: bad OCR or inattentiveness during tagging (should be re-tagged).

However, to validate consistency and completeness of tagging, and to guarantee the right text chunk was tagged, open each and every document to avoid cases like this. For example, the data value for the field invoice_number is "2574635", but if we open a document we'll see that "2574635" belongs to Reg.No and not to Invoice Number. The same problem can be with dates when there are some of them in the document (order date, delivery date, etc.) and only context allows us to check it's the right tagging.

tip

The following macros can be used to speed up the process of opening links. Select a range of links and run it.

Sub links_opening()
On Error Resume Next
For Each cell In Selection.SpecialCells(xlCellTypeVisible)
    ThisWorkbook.FollowHyperlink cell

Next cell
End Sub

This method of validation can be used when the speed of tagging is high or you have to deal with already-tagged large amount of data; it allows easier and faster processing of large-volume data sets and defines incorrect records that will be corrected or removed if the size of the data set allows.

tip

Combine all the batches that are ready for validation in one file to generate links to the tagged content not to run this BP for each batch.

Mistakes categorization

As it's impossible to correct tagging mistakes during validation in browser mode, WorkFusion recommends assigning each record a particular category based on the inconsistencies found: "Correct", "Bad OCR" and "Re-tag". Look through the points below to understand how to categorize different types of mistakes. Such categorization will allow us to be aware of the quantity and quality of documents which can be used in the training and test set.

Good

Absolutely correct and consistently tagged documents, in which no values are missed and everything tagged correctly; OCR is good.

For test only

Corner cases (Bad OCR, Rare Template) - documents, which are not eligible for training, but which exist in real life documents' flow and can be processed (at least partially).

These are documents for which data-values for ALL THE FIELDS presented in the original document (except handwritten) are correct, but:

  • they belong to rare template, there are less than 10 such documents and they significantly differ from the others (so the Model will not be able to be trained well on such a small number of examples).
  • or after OCR there are some issues: values with OCR mistakes, broken structure (so field can be tagged only with "Append selected to" option), context of fields has BAD OCR, etc.

For training only

Positions of presented tags are correct and tags are complete, but some tags are missed (usually due to OCR issue).

Also documents with wrong data-values or tag missed due to tagging mistake can be used for training, but only if there is no time or capacity for re-tagging. Normally it is better to re-tag such documents.

Re-tag

Document contains some issues with both data-values and tags, but it still can become "Good", "For training" or "For test" after re-tagging.

Good document but some values weren't tagged though presented in the document.

Complete and correct tagging with good OCR around the tagged values, but field is tagged in wrong place in different documents (inconstancy)

Value tagged from different locations across one template.

Documents which are tagged incorrectly, i.e. at least one of the following.
  1. Not completely

Tampa Consultancy Solutions Ltd.
Tampa Consultancy Solutions Ltd.

  1. Wrong value was tagged

Reg.No. 27638217 tagged as client_id.

  1. Wrong tag assigned to the value

angelina.bell@csn.com tagged as requester_soeid.

  1. Not all values of one field were tagged (for group fields)
ItemPriceTotal
Lenovo
A350		456.90456.90
HP
5389F		345.67345.67
Dell 351244.88244.88
Invoice Total1 047.54
  1. Bad OCR was tagged
Bank AccountSWIFT
Please remit your payment
to "H1rA~@Lt1."		B~40)ndg"! Reos Sth.
89OA~]c
Changed data value

Tagged string for product is "ALUMINIZED" so the data value should also be "ALUMINIZED", not "AL" or "Aluminium".

Exclude

None of the above mentioned.

Document has some considerable issue, so it is inappropriate neither for training or test sets. Usually it is documents with really bad original quality or with considerable OCR issue.

For example:

  • There is some serious OCR problems for all fields, nothing is recognized correctly.
  • Totally corrupted structure of document.
  • There are only handwritten fields in the document.

Enabling category dropdown

So as not to require typing a name every time, a drop-down function is recommended.

  1. Create an Excel copy of your CSV to keep all the formatting.
  2. Create a new spreadsheet, and insert categories.

  1. Select the range for which you want to set a drop-down list, click the Data tab, and then Data Validation.

  1. Select List option from the Allow: field, and then specify your range of possible values.

Validate via new manual task

This method can be used when the speed of tagging is rather low or the customer's document flow doesn't allow processing large amount of records at a time.

It allows the following:

  • Correcting mistakes during validation.
  • Checking tagging consistency, completeness and data values.
  • Not generating links to the tagged content.

However, when working on-site, a customer's limited server resources may considerably slow down Control Tower and WorkSpace.

Creating a new manual task

  1. Generate snapshot from the completed task.
  2. Create a new task by making an independent copy of the tagging task.
    1. As input data, upload the snapshot with the tagged data.
    2. On the Design tab, change the settings of the task.
    3. Click Show Advanced Options.
    4. In the Original Document field, provide the name of the column where the links to original data are stored.
    5. In Default Value, provide the name of the column in a snapshot where tagged data is stored. It should always be the same as Unique Code plus "_tagged" suffix.
  3. Click Save.
  4. Run the task.
  5. Go to Workspace and accept this task. If the record is correct, click Submit. If there are some inconsistencies, correct and submit the task.

Additional information

Despite the method of validation, the following issues must be observed:

  1. All the required fields are tagged. Here you need to make sure the "n/a" option is soundly chosen and the value is not missed due to inattentiveness — except for the cases when it's totally corrupted by OCR.
  2. If the value is presented in the document, it was tagged. Otherwise, missed fields will be badly represented and such documents should be re-tagged or the data set should be increased.
  3. Fields are tagged from the same location across the whole data set. If more than one variant of tagging the same field is found in one template, see how often this happens. If three times out of 200 documents, the supplier_name was tagged in some other place it’s better to exclude such documents.
  4. Pay attention to documents’ layouts as well. Note that due to OCR, the layouts of one vendor can be multiplied — for example, because of the table structure broken, etc. Such layouts, as well as badly represented layouts (fewer than 10 for a vendor), should be increased, if possible.
  5. Complete value is tagged.
  6. Data value was not changed manually — for example, the tagged string for a product is "ALUMINIZED", then the data value should also be "ALUMINIZED", not "AL" or "Aluminium".
  7. There is no tagging of corrupted values which cannot be restored.
  8. Option “Append selected to” was used correctly. If you are not sure, just try to select a desired value. If no extra information (except for the required text chunk) gets into your selection, there is no appended parts.

If the final number of good documents is not enough, it is highly recommended to correct a category "Re-Tag".

← Prepare for labelingSplit training and test sets →