Final dataset review
Goal: To preprocess gold data (normalize), if necessary, for model training and check label distribution.
Input: CSV file with validated and corrected documents.
Output: Number of values per field, gold data without extra spaces/new lines for model training.
Before splitting documents into test and training sets and running model training, it's necessary to double-check gold data. All the data values of the labeled documents must be final and, if necessary, pre-processed.
A Data Analyst has to make sure that values are:
- Well-represented
- Normalized
Label distribution
It's recommended to сheck the gold label representation each time when the validation of a batch is fulfilled. Before splitting into batches, the final check is to be conducted to ensure the same level of representativeness in training and test sets. That can be done with the help of a bot step.
Refer to label distribution check for more information.
In the result, one can notice that there is sufficient label distribution, but only half of all the documents contain lables, but still this number will be enough to train the model.
If the number of labels is few (less than 100) the data analyst has to escalate this, and request more documents, containing rare fields, and explain risks to the customer if there are not any more documents.
Example of email:
Gold data normalization
If necessary gold data should be normalized. Refer to gold data normalization for more information.
Format: CSV file.
Structure: Has column that contains labeled XML/HTML for normalization. Column name by default is tagged_text, it can be changed in BP on params step. Learn more about this bot here.
If there is extra logic that should be applied — for example, data values converting, mapping, removing additional symbols, etc. — extra rules should be designed by the Data Analyst and Machine Learning Engineer.