Test report and improvements
Goal: Analyze results of ML training and propose ways to improve them.
Input: Test set, trained model.
Output: ML results, post-processing rules, report with statistics.
At this stage, the Data Analyst needs to estimate model results and, if necessary, propose rules to improve them. The model is evaluated on the test set for each training iteration, if there are several of them, and DA calculates the statistics of each iteration based on evaluation results. DA needs to calculate the statistics for each iteration, then analyze delta for tagging iterations and model mistakes.
Data Analyst should identify reasons of model mistakes (consider all cases of FP, FP/FN results). Then propose options to improve ML results. Improvement options include:
- rules for post-processing and mapping rules defined by Data Analyst
- extra feature extractors developed by Machine Learning Engineer.
Data Analyst analyzes the statistics of each training iteration and statistics with post-processing. Then counts delta for ML results changing after training iterations and post-processing to define if the model performance improved or not.
When the defined rules are implemented by the ML Engineer, the Data Analyst calculates final statistics. In the end, Data Analyst's task is to provide the final statistics of model performance to the customer. All this data is compiled and submitted to the customer in the form of a report.
So, there are two main steps to this stage: analysis of results and report submission.