Skip to main content
Version: 10.2.8

Prepare labeling dataset

Dataset requirements​

There are four main requirements to dataset building you need to account for:

  • Size. The dataset size depends on the complexity and variability of documents and the number of particular documents available. Usually, the approach is to use a minimum-size dataset to achieve sufficient results. The average dataset size varies across use cases, most often from 300 to 500 documents.
  • Representativeness. The dataset should reproduce the document flow in production. This means the dataset should contain the same distribution of different document types and subtypes (usually referred to as layouts) at a smaller scale. Also, the dataset should provide enough documents for each layout.
  • Quality. This characteristic has two aspects. First, the tagged dataset should not contain any labeling mistakes—any mistakes produced by selecting incorrect values or selecting incorrectly. The second aspect is the OCR quality—it should be the best possible.
  • Structure. The dataset should be well-ordered and easy to manage. As a rule, this is achieved by splitting the dataset into batches, usually sorted by suppliers and layouts.

Dataset collection stages​

There are five stages of the dataset collection:

  1. Investigation and analysis. At this stage, input documents are analyzed. The DA estimates the document quality and distribution. Then, the DA and SME review the set of fields and align on the labeling logic. Later, the DA provides a labeling instruction that includes all rules and corner cases based on the alignment results. The labeling instructions contain information about the business and labeling logic: what should be tagged as correct answers, what's most important, and where exactly in the document it should be tagged. Such instruction should be maintained during labeling and validation.
  2. Conversion. To train a model, the DA needs all documents in different formats to be converted into XML or HTML format. Usually, documents are digitized from PDF or image format and converted into XML or HTML one using the OCR engine. It takes documents in the PDF format as the input and produces the same documents in XML or HTML format as the output.
  3. Training. Training is dedicated to acquiring skills required to label documents quickly and effectively. The DA provides a training task to the SME team and gives a presentation on how to label correctly, then validates the results, and provides feedback.
  4. Labeling. This term can refer to the labeling process in any automation use case type. However, most often, labeling is used for Information Extraction use cases. This is the process of highlighting all available fields in a batch of documents to supply them for model training. Usually, this is the longest and most important stage of the dataset collection. A Manual Task is created, with all the fields that need to be extracted, and all the documents in XML or HTML format are uploaded there. The SME team starts working on them in the Workspace application.
  5. Validation. When documents are tagged, the DA starts validating the tagged text. The primary purpose is to ensure there are no labeling mistakes or omitted fields, and values are labeled consistently. This stage can start right after the labeling begins, and quite often, the DA and SME teams work in parallel. The final output is a dataset referred to as Gold Data for model training, which means the model takes all the fields as 100% correct values and reproduces them in production. It's essential to ensure dataset quality because a model can provide the best results only on accurately and precisely tagged documents.

When the dataset is collected and checked, the DA splits it into two parts:

  • A training set, which is used for training the ML model on gold values. For Information Extraction, the gold values are a set of fields to be extracted. For Classification, documents are accompanied by a target class. The training set makes 80% of the entire dataset.
  • A test set, which includes unseen documents used for evaluating the ML performance and testing for possible exception cases in production. The test set composes 20% of the entire dataset and is not used for model training.

Next step​

Next, explore the labeling process.