Design Manual Task for labeling

Goal of the step: design a Manual Task, accounting for the business logic of all documents and fields

Output of the step: a functional Manual Task you can use for labeling.

You can create a Manual Task for one of the following most common use cases:

• Information extraction (IE)
• Classification

For model training, find and label entities (fields) in an XML document. They are to become values for the model to train on.

To create a quality Manual Task, analyze the logic behind the documents, document types, and their layouts in each use case. Pay special attention to the field logic and appearance in IE cases. The data analyst reviews the set of fields in a document and defines which fields have only one value and which have several of them—in other words, identifies a field as a single-value or multi-value one.

A single-value field is a field that has only one result value in each document. Typical examples are supplier_name, total, date.

A multi-value field is a field that can have multiple result values. Typical examples are product_name, product_price, product_quantity, and so on.

note

It's necessary to identify clearly in advance single-value and multi-value fields and all cases for each of the types. Otherwise, in later stages, extra effort will be required to redesign a Manual Task and retrain the model.

Required or optional

A required field is a field present in all documents from all batches. For example, invoice_number, invoice_date, and so on.

An optional field is a field present only in certain documents or in specific batches. These fields usually have special additional logic.

Required and optional fields need to be considered together with SMEs. However, if the number of fields is 10 or higher, it's strongly recommended to make it impossible to submit a task without validating that all necessary values are labeled.

Best practices

Apply hot keys

Hot keys are helpful for quick labeling. To increase speed, assign each field a hotkey and teach SMEs how to use them.

There are two approaches to assigning convenient hot keys:

• Assign intuitively understandable and memorable letters, for example, the first letters of fields: d = date, p = price, n = invoice number, and so on.
• Assign the letters and figures that are close to each other on the keyboard: 1, 2, 3 = first three fields; q, w, e = another three fields; a, s, d = yet another three fields; and so on.

Use N/A

It's better to mark optional fields as required and make a N/A selection available to minimize SMEs' mistakes due to inattentiveness.

Split long entities

Check whether a value can be split into independent parts, for example, the entire address line can be very difficult to extract as address formats usually differ from country to country or even within different document templates.

It's recommended to split the value into shorter entities located within the same XML tree element: street address, city, state, postal or zip code, and so on.

caution

Be careful with value splitting. For instance, you cannot split values, such as company_name as their parts are not independent, and the value only makes sense with the whole name.

Enable additional fields

If necessary, for a more informative output, you can enable additional fields, such as:

• A drop-down list to classify documents: for example, if you have a combined dataset in five different languages, add a special field with a drop-down list of the languages to simplify the further analysis of the dataset and its grouping.
• A checkbox to mark bad OCR: even after OCR tuning, there can still be documents with bad OCR. To avoid labeling such documents and easily find them later, it's recommended to include a special checkbox for OCR.

Avoid making Manual Task too big

It is recommended to include 7–10 fields in a single Manual Task. If an AI Agent implies the extraction of a larger number of fields, it's better to create several tasks.

For example, if you need to extract 30 fields, create three separate tasks with three different sets of fields in each. This way, your dataset is processed three times through three different Manual Tasks.

Otherwise, too many fields in a task may lead to lower concentration and, consequently, lower quality of the dataset.