Edit

Prepare for labeling

Goal: Train SMEs to tag the documents properly to ensure high-quality data set and to prepare documents for tagging

Input: Original documents in XML/HTML format, SMEs team

Output:

• Documents prepared for tagging: split into batches, with tagging logic predefined.
• SMEs trained to tag documents qualitatively.

This stage is critically important, because here the Data Analyst lays the foundation for the whole data set collection process. If there are any issues at this stage not revealed and addressed properly, they can result in major problems during later stages.

The main aim of this stage is to prepare everything for tagging to start: to train and qualify SMEs, and to prepare documents for tagging. SMEs' training and qualifying means that they are able to tag documents and meet the requirements of a high-quality data set and documents preparation — including splitting into batches, establishing tagging logic and describing it in tagging instructions.

DA needs to perform the following steps in this stage:

1. Manual task design
2. Train SMEs
3. Split into batches
4. Tagging logic definition
5. Tagging instructions provision

In the end of this stage, the Data Analyst assigns a specific manual task for each SME, provides a batch and instructions for it, and creates a data tagging tracker to track SMEs' speed of tagging.

In the result of this stage, the batches and manual tasks are prepared and assigned to SMEs, a report on the whole stage is provided, and all the necessary preparation for the first tagging iteration is conducted.

The following template can be downloaded and used as a report for this stage:

• Download report example

Risks at the stage:

1. Wrongly configured Manual Task
   The Manual Task should be configured according to business logic of all the documents. It should have correct Answer Types and Grouping, because if a mistake is revealed in the process of tagging, it will take extra time and effort to reconfigure the manual task, re-tag documents, and even retrain the model (if training iterations start in parallel with tagging).
2. Low tagging quality
   There is a risk to involve under-skilled SMEs for tagging, which could result in inconsistent and incorrect tagging — which would result in bad quality for the data set or extra time to re-tag and verify documents. SMEs must be trained well or at least pass through the Qualification task after additional training.