next

Edit

Prepare data set

Data set requirements

There are four main requirements we need to fulfill to build the data set.

  1. Size. The size of the data set depends on documents' complexity and variability, and the number of particular documents available. Usually, the approach is to use the data set at a minimum size to achieve sufficient results. The average size of the data set varies among use cases and often numbers 300-500 documents.
  2. Representativeness. The main rule applied is that the data set should reproduce the document flow in production. This means the data set should contain the same distribution of different documents' types and subtypes (usually referred to as layouts), at a smaller scale. Also, the data set should provide enough documents for each layout.
  3. Quality. This characteristic has two aspects. First, the tagged data set should not contain any tagging mistakes - any mistakes produced by a selection of incorrect values or in an incorrect fashion. The second aspect is the OCR quality of the documents, which should be at maximum possible OCR quality.
  4. Structuredness. The data set should be well-ordered and easy to manage. As a rule, this is achieved by splitting the data set into batches, usually sorted by suppliers and layouts.

Stages of data set collection

Then, there are five stages of the data set collection.

  1. Investigation/Analysis. At this stage, the analysis of the input documents is done. DA estimates document quality and distribution. Then, DA and SME review the set of fields, and together, they align on tagging logic. Later, DA provides a tagging instruction that includes all the rules and corner cases and reflects this alignment. Tagging instructions contain information about business and tagging logic: what should be tagged as correct answers, what's the most important, and where directly in the document it should be tagged. Such instruction should be maintained during tagging and validation.
  2. Conversion. To train a model, DA needs all documents in different formats to be converted into XML or HTML format. Usually, documents are digitized from PDF or image format and converted into XML/HTML format with the help of the OCR tool by ABBYY. It takes documents in PDF format as input and releases the same documents in XML/HTML format as output.
  3. Training. Training is dedicated to acquiring the necessary skills to tag documents quickly and effectively. Data Analyst provides a training task to the team of SMEs and gives a presentation on how to tag correctly, then validates the result and provides feedback.
  4. Tagging. This term can refer to labeling for all types of Use Cases. But the term tagging is used most often for Information Extraction Use Cases. This is the process of highlighting all the available fields in the batch of documents to supply them for model training. Usually, this is the longest and most important stage of the data set collection. The Manual Task, with all the fields that need to be extracted, is created and all the documents in XML/HTML format are uploaded there, and theSME team starts working on them in WorkFusion's application WorkSpace.
  5. Validation. When documents are tagged, DA starts validation of the tagged text. The primary purpose is to ensure there are no tagging mistakes nor omitted fields and values are labeled consistently. This stage can start right after tagging begins and quite often the DA and SME team work in parallel. The final output is the data set, which is referred to as Gold Data for model training, which means the model takes all the fields as 100% correct values and then will reproduce them in production. That is why it's so essential to ensure data set quality - because the best ML results can be achieved only on accurately and precisely tagged documents.

When the data set is collected and checked, Data Analyst splits it into two parts: a training set and a test set. We use the training set for training the ML model on gold values. For Information Extraction, the gold values are a set of fields that should be extracted; for Classification, documents are accompanied by a target class. The training set composes 80% of the whole data set. The test set is unseen documents used for evaluating the ML performance, and to test it for possible exception cases in production. The test set composes 20% of the whole data set and is not used for model training.

Move further

Next, we'll explore the tagging process.

← Study documentsStart tagging →