Learn about labeling workflow
Roles
The labeling flow includes the two main roles: Data Analyst (DA) and Subject-Matter Expert (SME). Though they have different functions in the automation delivery process, they work closely together. In particular, they collaborate to collect a dataset: the SME shares their expert knowledge, and the DA delivers the dataset. For specific differences, see the table below.
| DA | SME |
|---|---|
The role is essential for unstructured data processing. The DA is a person assigned to help with process automation via Work.AI in terms of document workflow. Their main goal is to collect a high-quality dataset to train a model. The DA trains other people how to label documents in Workspace, verifies the quality of the documents for model training, and validates the result. | This is an individual (on the customer’s side) with expert knowledge of a particular process, function, technology, machine, material, or type of equipment who assists in understanding of business rules for automation projects and participates in data labeling. The SME provides information on the documentation logic, meaning, workflow, and any corner cases. They learn how to use Workspace with the help of the DA and label documents in the application to obtain a dataset. |
Functions
Now, let's talk about the particular functions DAs and SMEs fulfill.
| Dataset collection steps | Data Analysts | Subject-Matter Experts |
|---|---|---|
| 1. Investigation | Study the document logic, quality, and distribution | Provide document samples |
| 2. Alignment on labeling logic | Create labeling instructions | Share expert knowledge on the logic of fields and specific corner cases |
| 3. OCR | Check document quality after OCRing | Search for additional documents if necessary (with better quality for OCR) |
| 4. Training | Provide one or more training tasks and teach how to use Workspace | Take up the training task in Workspace |
| 5. Active labeling | Validate labeled data | Label documents using the instructions and guidance by the DA |
Workflow
As you now know the roles and functions of the DA and SME, it is time to learn the workflow of their collaboration.
| # | Step | Description |
|---|---|---|
| 1. | The DA studies requirements. | In the first step, the DA should learn as much as possible about the documentation logic and workflow, document layouts, and production distribution. The DA reviews the requirements for the automation solution, document quality, format, and data that needs to be selected in the document in terms of fields. |
| 2. | The SME shares samples and specific values. | The SME consults with the DA on different questions about the documents and explains the logic of document samples. |
| 3. | The DA and SME hold a questions-and-answers session and align on the labeling logic. | The DA reviews the received documents and makes notes for the SME about the document structure and quality. This investigation step can be a reiterative series of questions and answers, resulting in a common understanding of what values should be labeled and where they should be found in documents of different types and vendors. It's crucial to cover all the documents, formats, and templates to be used in a dataset. |
| 4. | The DA splits documents into batches. | The step is optional. If the amount of documents is overwhelming or you are not sure about the success of the initial labeling, divide available documents into batches. |
| 5. | The DA prepares labeling instructions and a Manual Task. | The DA records the labeling logic established at the questions-and-answers session in the form of instructions for the SME and designs a labeling Manual Task. It's essential that labeling instructions are comprehensive and followed by the SME while labeling. Any further nuances discovered later or new documents added to a dataset can entail a review of the entire dataset, re-labeling, or redesign of the Manual Task. That's why it's crucial to cover all the documents at Step 3. |
| 6. | The SME has training on labeling. | The DA provides training tasks for SMEs. Labeling rules are explained there, and experts study how to work with the Workspace application. Labeling rules and typical mistakes are discussed when the training task is outlined. |
| 7. | The DA provides feedback. | The DA provides feedback on the results and shows what can be improved. When enough training is done, labeling for dataset collection begins. |
| 8. | The SME labels documents. | A Manual Task is created with a set of necessary fields. The DA uploads documents, and the SME starts labeling in Workspace using the instructions from the DA. |
| 9. | The DA validates the results. | This is a cycle: the SME labels documents and the DA validates the results. |
This is a visualization of the described workflow:
Color coding:
- Pale orange is an SME function.
- Light blue is a DA function.
- White is for collaboration.
The order of the steps is not strict, but it is highly recommended. Mind that the DA can modify some steps.
Next step
Now that you know about the DA and SME roles, it's time to learn about the technology. The next module describes the automation solution components that labeling relies on.