Skip to main content
Version: 10.3

Report

Goal: Report the ML quality to the customer and communicate poor results, if any, in an accurate yet positive way, and propose further solutions.

Output: Excel file with report.

Ways to report ML quality​

There are several ways to demonstrate ML quality to the customer, some of which can be combined:

  • Show the average document-level statistics in a deck.
  • Provide a table with field-level statistics.
  • Launch the documents from the test set (tagged by the model) for SMEs to review in a Manual Task.
  • Send the statistics of extracted data to the customer so they can also calculate the statistics from their side.

WorkFusion recommends using a report that includes most of these methods.

Report​

Metrics​

Some metrics are common to and some are different for information extraction (IE) and classification.

TP (True Positive)TN (True Negative)FP (False Positive)FN (False Negative)FP/FN (False Positive/False Negative)
  • Extracted (TP + FP + FP/FN)
    The number of objects/instances of a specific field that were extracted by the model. The number of extracted fields can be fewer than the number of documents, if the field does not appear in all documents and because the model may miss some values. If the field is multi-value, then the number of extracted fields can be more than the number of documents.
  • Gold (TP + FN + FP/FN)
    The number of objects that are available in the test set. "Gold" in this case means the fields that were tagged by SMEs and thus are considered 100% correct.
  • Correct (TP)
    The number of objects that were correctly extracted by the model. This number doesn't equal the number of extracted fields, because not all of the extracted fields might be correct.
  • Accuracy (Correct/Extracted)  A metric that shows how exact the model is in its decisions. It is calculated by dividing the number of the values extracted correctly by the number of all values extracted.
  • Automation Rate (Correct/Gold)   A metric that shows how complete the model is in its decisions. It is calculated by dividing the number of correct values extracted by machine by the number of all values available/extracted by SMEs. This metric shows how much of work the model is able to automate, while precision shows how accurately it is done.
  • F1 (2*(Accuracy*Automation Rate))/(Accuracy+Automation Rate)
    Score is the harmonic average of the Automation Rate and Accuracy.
  • Delta of Automation Rate and Accuracy per iterationThese metrics are the most commonly used, but extra metrics can be added as well.

Structure​

The report is an Excel file that includes three main information blocks:

  • Deliverables should describe the following:
    • Model info: name, type, version (extraction or classification, to distinguish between trained models when some changes were applied).
    • Field names: names of the fields for which the model was trained.
    • Training and test set size.
  • ML statistics:
    • Document identifier (original_document_link)
    • Gold value (document_link_tagged)
    • Results extracted by the model (document_link_extracted)
    • Gold vs. extracted value per field and per document
    • Decisions for the gold vs. extracted comparison: (true positive (TP), false positive (FP), true negative (TN), false negative (FN), false positive/false negative (FP/FN))
  • Improvements:
    • Model mistakes that were not processed, with the reasons and solutions offered
  • Additional:
    • Pivot table with documents per vendor in test and training sets and their OCR quality
    • Number of documents per class in test and training sets and their OCR quality
    • Extra metrics communicated with customer
note

The model is focused on tagged strings, and the statistics are calculated and report is provided for data values (as the customer will see data values).

Results and explanation​

Statistics should be calculated after each change in automatic data processing implementation (new model version, next iteration of training, post-processing application, and so on). The final presentation of automation results should be shared with the customer together with the reasons why some stats on some fields fall behind success criteria, if there are any. Moreover, further steps and solutions should be proposed.

For example, out-of-scope cases:

  • Handwritten text
  • Bad original document quality, which resulted in bad OCRed document quality
  • Bad field representation

The issues mentioned above should NOT be reported at the last stage for the FIRST time. They should be raised to the Delivery Manager and discussed with the customer during previous steps.

Actions should be proposed for the next phase or future project to build on the current achievements and improve to desired level. For example, an agreement can be made with the customer that badly-represented fields can be excluded from model stats calculation until their number in the dataset reaches a desired level.

Download a report example.