Test report and improvements
Goal: Analyze results of ML training and propose ways to improve them.
Input: Test set, trained model.
Output: ML results, post-processing rules, report with statistics.
At this stage, the Data Analyst (DA) needs to estimate model results and, if necessary, propose rules to improve them. The model is evaluated on the test set for each training iteration, if there are several of them, and the DA calculates the statistics of each iteration based on evaluation results. The DA needs to calculate the statistics for each iteration, then analyze delta for labeling iterations and model mistakes.
The DA should identify the reasons for model mistakes (consider all cases of FP and FP/FN results). Then, propose options to improve the ML results. Improvement options include:
- Rules for post-processing and mapping the rules defined by the Data Analyst
- Extra feature extractors developed by machine learning engineers
The DA analyzes the statistics of each training iteration and statistics with post-processing. Then, it counts the delta for the ML results changing after training iterations and post-processing to define if the model performance improved or not.
When the defined rules are implemented by the ML engineer, the Data Analyst calculates final statistics. In the end, the DA's task is to provide the final model performance statistics to the customer. All the data is compiled and submitted to the customer in the form of a report.
So, the step comprises two stages: