Technology. Analyze ML model statistics

In this section, you will learn how to analyze Business Process executions with legacy Tableau-based analytics to identify gaps in productivity and understand possible bottlenecks.

Understand Machine Learning basics

WorkFusion recognizes two main cases where AutoML can be effectively applied:

• Classification
• Information extraction

In this chapter, you will get acquainted with the basic metrics of classification and information extraction models. Also, you will try to understand how to identify the issue and what could be the cause of the bottleneck.

Classification

The classification case is applied when it is necessary to define the class for an item (document). Usually, each class is associated with a specific document type. For example, invoices, purchase orders, and claims are processed in one workflow, but each document type is handled differently. That means you need to classify these documents before applying automation.

Information Extraction

The information extraction (IE) case is applied when the data defined by the business logic is extracted from documents and processed according to business rules. In terms of information extraction, each data point is referred to as a field. For example, the invoice number, supplier name, number of products, and total amount have to be extracted from all invoices.

Model metrics

Accuracy is the most intuitive performance measure, and it is the ratio of correctly predicted observations to total observations. In other words, it is the total number of correctly classified or extracted objects, divided by all values classified or extracted by the machine.

Automation Rate reflects how many correct results the machine found out of total correct results. In other words, it is the number of correctly classified or extracted objects, divided by all the objects of the type existing in the dataset (confirmed by SMEs).

Rework reflects the amount of effort required from a person to correct machine errors. These are the values that were classified or extracted incorrectly, divided by all the correct values that are present in the dataset (confirmed by SMEs).

Automation Efficiency indicates the total amount of work automated by the machine. It is calculated as the amount of useful work done by the machine. Automation Efficiency = Automation Rate – Rework.

STP (Straight-through Processing) is the rate of documents processed entirely by bots without manual effort. STP Rate equals the number of documents processed with bot assistance only, divided by total documents.

Find bottlenecks

As an Operations Manager, you can face the following challenges:

• The amount of time that users spend on tasks has increased without significant changes in the document volume or process changes.
• Users noticed that, in a Manual Task, several fields are almost always not filled or incorrectly tagged and need to be manually processed.
• Users noticed that the number of tasks for manual processing has increased.
• You want to find opportunities to increase the automation rate.

To understand the cause of the problem, you have the AutoML dashboard to help you understand whether this is a model issue or not. Below are the potential deviations that you can notice while viewing the dashboard:

• STP rate goes down

  The chart provides data on the STP metrics—the rate of documents processed entirely by bots without manual effort.

  In the sample chart below, you can see that, on August 10, the STP rate fell for some reason.

  

• A high percentage of Fully Manual or Bot Assisted work

  The Volume chart shows how many documents are processed daily.

  The execution type splits the processed volume:

  • Automated: a model fully processed the document.

  • Bot-assisted: a model extracted fields partially.

  • Fully Manual: the document was handled manually without a model.

    In the sample chart below, you can see that, on average, the volume of documents has not changed on August 10, but the number of documents processed fully manually has increased.

    

• Manual Effort goes up

  

  This chart shows the total amount of manual effort spent on document processing, both fully manual and bot-assisted processing.

  Manual Effort (in hours) equal to the real time spent to perform a task: the time spent manually extracting the fields the model failed to extract.

  From the sample Manual Effort chart, you can conclude that, on August 10, users spent significantly more time on tasks than in previous days.

---

Generally, changes in the charts above are interrelated. For example, if the STP goes down, then the number of manually processed documents increases. Hence, the manual effort also increases. The first three charts help determine where and when any changes occurred in the process. Next, you can go down to the level of model results where you can trace what specific fields and values are involved.

However, it is also possible that the STP rate remains at the same level. Still, the number of documents increases significantly (for example, another source is added), and therefore the amount of manual work grows. However, this is not due to the deterioration of models, which you can conclude based on the graphs below.

• A high percentage of mistakes / low confidence

  

  Quality description metrics are as follows:

  • Not Learned is the ratio of fields that should have been but were not extracted by the model.
  • Low Confidence is the ratio of fields that should or should not be extracted but were extracted, with the model score lower than the selected Accuracy Threshold.
  • Mistake is the ratio of fields that should and should not be extracted, but were extracted with wrong values, and the model score is equals or higher than the selected Accuracy Threshold.
  • Correct is the ratio of the correctly extracted fields.

  The Automation Rate chart has the following metrics:

  • Automated is the ratio of documents processed successfully by the model with or without human assistance.
  • Fully Manual is the ratio of documents processed by humans.

  Using the filters on the dashboard, you can see what metrics were relevant for the past period. You can assume that over the past week, the number of Correct answers and the Automation Rate were higher. In the chart above, you can see that 7.56% are Mistake and Not Learned values. So, you need to go down further and see what fields have difficulties for now.

• A high percentage of field appearances in Manual Tasks

  

  The AutoML Statistics by Field chart shows the model quality results per each field across all documents.

  Set the Model Type filter to Classification or Extraction.

  

  In this example, you can see that the vendor_address field has a large percentage of Mistakes. In some cases, it was not recognized by the model at all. In only 48.39%, the values are Correct. The answer why this could happen will solve the issue.

  The AutoML Statistics by Field table includes the following metrics:

  • Accuracy means the total number of correctly extracted vendor_address values, divided by all values extracted by the information extraction model. Not Learned values are excluded from the metric as they were not extracted.

  • Automation Rate is calculated as correctly extracted vendor_address values, divided by all correct values existing in the documents.

  • Rework is the number of values that were extracted incorrectly, divided by all the correct values in the documents.

  • Automation Efficiency is equal to Automation Rate minus Rework.

  Fields with the Mistake or Not Learned values are both the cause of poor results and a point for improving automation performance. By analyzing the detailed field chart, the following issues can be identified in most common scenarios:

  • Model results: Mistake or Low Confidence

    Possible reason	Solution
    Training and test datasets prepared with inaccuracies, or values are missing	Retrain the model.
    Inconsistent gold data or insufficient normalization (extra symbols, different data types)	Retrain the model. Apply post-processing.
    Incorrect grouping for multi-value fields	Apply post-processing.
    Specific or broken document structure after OCR	Apply post-processing. Use additional components (ML SDK).

  • Model results: Not Learned

    Possible reason	Solution
    Not enough examples in the training set.	Retrain the model. Use additional components (ML SDK).
    Tagging inconsistency in the training set. The field is tagged in different positions.	Retrain the model.
    New document structure after OCR within the known layout.	Retrain the model.
    Complicated dependencies in extracting logic that the out-of-the-box model failed to detect.	Use additional components (ML SDK).

Summary

The legacy AutoML dashboard is handy for a general analysis of ML model performance and to explain why the model works in one way or another. It also allows you to conduct a detailed analysis of the results and make decisions on improving the model or detect that the automation maximum for a Business Process is reached.

To summarize:

• Classification of documents and extraction of information from them are the main AutoML use cases.
• Such metrics as increased time spent on a task or increased amount of manual work indicate a bottleneck.
• To detect a bottleneck, use the AutoML dashboard with the STP, Volume, and Manual Efforts charts.
• Also, remember to check Model results and analyze the percentage, **Errors, and field appearance in Manual Tasks.