Automation quality metrics

Automation Accuracy

Information extraction problems:

For Information Extraction, Automation Accuracy equals Precision in the data science terminology. It reflects how precise the extraction is, and hence is calculated as the ratio of correctly extracted objects, i.e., TP (meaning True Positive), to all the extracted objects, i.e., TP + FP (meaning False Positive).
Classification problems:

For Classification problems, Automation Accuracy equals Accuracy in the data science terminology. It reflects how accurate the classification is, and hence is calculated as the ratio of correctly classified objects to ALL the classified objects.

Automation Rate

In the data science terminology: recall. Automation Rate reflects how many correct results the Machine found from ALL the correct results (mean Gold). Automation Rate is calculated as the ratio of correctly classified or extracted objects (Correct or TP) to all the objects of this type existing in the data set (Gold or TP + FN).

Rework

If Automation Accuracy is 100%, Automation Rate would reflect Automation Efficiency. However, Machine does make errors that need to be corrected by Person. Rework reflects the amount of effort required from Person to correct Machine Errors. This metric is calculated as the ratio of the values that were extracted incorrectly (Precision mistakes or FP) to all the correct values that are present in the data set (Gold).

Rework may be MORE THAN 100% in case of too many FPs. See the example below.

Automation Efficiency

Automation Efficiency indicates the total amount of work that was automated by Machine and could be calculated as the amount of useful work done by the Machine, i.e., Automation Rate, less the amount of errors made by the Machine, i.e., Rework:

Automation Efficiency may be NEGATIVE, in case of Rework > Automation Rate. See the example below.

In a general case, where the Business Process may contain multiple steps and each step requires multiple object types to be extracted/classified, Efficiency for the whole automation step or the whole BP can be calculated as the ratio of completed effective work (the sum of useful work less rework at each sub-step, field, document type) to all the work that should have been automated (the sum of Gold values at each sub-step, field, document type).

Examples of metrics calculation

The BP contains two automation steps:

Classification for two document types: invoice, not invoice
Information Extraction from invoices

The number of input documents is 1,000.

Classification:

invoice	not invoice
500	500

Information Extraction, Invoices:

Field name	Number of gold values
`invoice_number`	500

Automation results

Classification:

Document Type	Gold	Classified	Correct (TP)
invoice	500	550	400
not invoice	500	450	300

Classified	Gold	Correct	Automation Accuracy	Automation Rate	Rework	Automation Efficiency

1000	1000	700

Information Extraction, Invoices:

Field name	Extracted	Gold	Correct	Automation Accuracy	Automation Rate	Rework	Automation Efficiency

`invoice_number`	400	500	350

BP Automation Efficiency:

Advanced example of metrics calculation:

The BP contains three automation steps:

Classification for three document types: cancellations of insurance type, new or renewal of insurance type, not insurance.
Information Extraction from cancellations of insurance type.
Information Extraction from new or renewal of insurance type.

The number of input documents = 2000.