Reading Model statistics
DM must be able to read model statistics. To make use of this guide, make sure that you are familiar with: (1) the notions of true positive (TP), false positive (FP), true negative (TN), false negative (FN), precision and recall - you can find them described in Data Analyst documentation; (2) mapping of this terminology to customer-friendly names.
All the tools of presenting model statistics that are describe below follow common structure: they contain data on statistics per field (accuracy and automation rate, result type (TP, TN, FP, FN)) and detailed gold vs extracted comparison (usually including the gold and extracted values themselves, result type for each and some ID of the record/document where they were observed). Statistics per field is what is usually presented to customer and also the first level at which you view the model results. Once you identify the problematic fields based on these stats, you can go deeper into specific gold vs extracted comparison to understand the reasons of such results.
For example, you observe low precision or recall for some field. You can find the corresponding columns in the detailed results and look through the error types to see which ones occur most frequently: did the model extract a lot of values incorrectly (FP) or did it miss many values (FN)? Further you can look at the extracted value itself and compare it to the gold value to see what caused the mistake. If extracted fields are counted as false positives, is it because the gold is missing or it is different from the extracted value? If missing, is it due to inattentiveness of SME or it indeed should not have been tagged? If extracted is different from gold, understand what the difference is: number written with/without comma, date in different format - such differences are fixed by post-processing and normalization; value completely different - might be due to incorrect gold or the field is complicated for the model or represented not well enough.
Check other typical kinds of mistakes in Interpreting results.
Automation dashboards
Dashboards are a convenient and user friendly way to present and read model statistics. Information contained in them is calculated based on processes where both model extraction was applied (from which we get extracted values) and human workers/SMEs submitted human tasks (from that we get gold values).
Note that the dashboard is filled in only if Automation Use Case containing "Statistics calculation" step is used in Automation BP.
AutoML dashboard WF 9.1
The AutoML dashboard contains analytics on manual work reduction resulting from automation, ML performance forecast, and statistics of the models. All the information can be filtered by process name and specific instance/execution of this process, as well as process status and execution time range.
In the AutoML Statistics by Field chart, you can find results for each field. Black (66.67% for order_total) is Correct/TP; orange (16.67%) is Mistake/FP; grey (16.67%) is Not Learned/FN.
The Gold vs Extracted Statistics chart can be filtered by process name and execution and field name. In addition, Gold vs Extracted Statistics allows to filter by:
- Values: all, incorrect only (FP+FN), correct only (TP+TN). By default this filter is set to "Incorrect only".
- Result type: in the dashboard for the purpose of being easy to read by customer FP is referred to as "Extracted with Errors"; FN as "Failed to Extract"; TP as "Correctly Extracted", TN as "Should not be Extracted".
- Confidence: the confidence with which a field was extracted is reported by the model
Apart from field name, confidence and result type, the table contains:
- Gold values: the values that were extracted by SME/human worker in the manual task and is supposed to be correct
- Extracted values: the value extracted by the model
- Documents: # documents in which this combination of gold/extracted values occurred.
At the bottom of the report you can see the number of expired human tasks (4 in this example). It is mentioned in the report, because they are excluded from the calculation of statistics.
Cognitive Automation Breakdown Dashboard WF 8.5
From version 8.5 till version 9.1 you can use the Cognitive Automation Breakdown dashboard to read model stats.
In this dashboard, once you choose a specific business process, you will be able to see Statistics per Field and Gold vs. Extracted charts.
In the Statistics per Field chart, you can find results for each field; specifically, accuracy and automation rate. Result type contains colour-coded details on TP (Correctly extracted), FP (Extracted with errors) and FN (Not extracted). For example, on the image below you can see that invoice_amount has 97.75% of TP (green), while the remaining mistakes are somewhat equally divided into FP (orange) and FN (grey).
The Gold vs. Extracted chart can be used for analysis of the model results and their reasons. It contains:
- Gold Value: value tagged by Human.
- Extracted Value: value that was extracted by model.
- Result Type (TP, FN, TN or FP, FN): result of Gold and Extracted Values comparison.
- Score: the number assigned by the model to each value to indicate how well it fits the criteria to be extracted.
- Confidence: the level of confidence of the model in that the score it assigned to the value or the decision it made is correct.
This information is listed for each field, accompanied with the name of the field, uuid of the task and uuid of the document where this result was collected from.
Excel report
Another commonly used way to report model statistics (especially for WF versions before 9.1) i s via Excel spreadsheet. Such report is described in DA methodology and in the project it will be prepared by DA, if needed with the help of MLE.
Model statistics are presented in the first tab of the report. To learn how to read them, refer to the page Interpreting results.
Precision and recall are related to each other: the higher precision threshold is set for a trained model, the lower recall will be as the model would be able to extract fewer fields at the desired level of confidence in their correctness. Vice versa, if smaller precision is sufficient, the model would consider more fields to be correct with less confidence in that decision.
The second tab of the report contains the detailed results based on which the statistics in the first tab were calculated.