Skip to main content
Version: 10.2.8

Success criteria

All requirements should be analyzed in terms of success criteria or the standards by which the team and the client can judge whether an automation goal is achieved. Success criteria are often linked to metrics.

Success criteria​

On the high-level, project success can be identified as:

  • Achievement of project objectives (quality, time, and costs)
  • Client satisfaction (delivered solution is sustainable, profitable, and meets the technical specification and functional requirements)
  • Project maintenance (the project is completed on time, at a specified quality standard, without disputes or safety issues, and within the budget)

Success criteria are usually manipulated to prove that the automation target is achieved (in exact numbers) and that automation generally brings value.

Overall success criteria are needed for the following:

  • Performance measurement. Indicate how well a process works in exact numbers, indicators, and benchmarks.
  • Decision making. Support evidence-informed evaluation and decision making, in particular formal adoption of the final solution.
  • Focus. Give focus on development and prioritizing.
  • Tracking. Measure progress and identify areas for improvement and development.

Success criteria should assess the move from where we are to where we want to be rather than maintaining where we currently are. Thus, success criteria set a standard or a level of achievement, a cut-off point at or above which success is assumed and below which success is not achieved.

Additionally, success criteria help to acknowledge the changes and improvements in the automated process and contribute to transparency and accountability.

Metrics​

Clients usually have their own ideas of value and success. You need to possess a high-level understanding of success, its characteristics, and a uniform definition in exact metrics. Metrics derived from success criteria help measure project success, align, and ensure you speak the same language with the client when you say something is successfully delivered.

Success criteria are often linked to changes, developments, innovations, plans, and so on. They set standards that play themselves as metrics or acceptance criteria. In general, success criteria define standards, and metrics help to identify these standards precisely. For example, the main success criterium of the project is processing time reduction. This success criterium is decomposed into metrics, which identify time reduced on document processing end-to-end, time spent on data entry, number of exceptions handled manually, and so on. Each metric should have its corresponding current (as is) and target (after automation) value to measure the progress.

Success criteria established in metrics should meet the following requirements:

  • Be achievable or deliverable within defined conditions (time, budget, team, technology, and so on).
  • Be clearly stated (exact numbers, formulas, definitions, and so on).
  • Be measurable (a clear set of actions defined to measure success criteria value).

If the conditions above are not met, you find yourself in a situation where you cannot prove solution success because success criteria are vague and can be manipulated, or success criteria are clearly defined, but there are no initial values to compare against or the process of measurement a metric is unclear, and so on.

Typically, success criteria can be expressed in terms of ML statistics stated as Accuracy and Automation Rate on a field level or document level. Clients can also be interested in FTE savings and overall effort reduction of the process. There can be also specific metrics for use cases, for example, maintaining increased document volumes with sources of the same team.

Generally, metrics are grouped into the following categories:

  • Time based: time reduction for end-to-end processing, time reduction for manual processing, target time to process one record in a Manual Task, and so on.
  • Money and effort based: FTEs saved, ROI, reduced costs due to an error, and so on.
  • Throughput based: number of documents processed during a defined period of time, peak volumes.
  • ML metrics based: Automation Rate and Accuracy.
  • STP based: % of records processed end-to-end without a manual touch.

The most common are ML metrics and STP. STP is Straight Through Processing indicator showing how many documents are fully automated in the workflow. Basically, an STP document is the one that has all the required fields extracted above the target accuracy threshold, which is not always feasible. Often, clients are focused on STP only, while this metric can be elusive due to document quality or other factors. Therefore, you should carefully select metrics to visualize all the benefits of automation, which may be far more than STP.

Among the factors that influence ML metrics and STP are as follows:

  • OCR quality: errors in fields due to recognition, complex unrecognized structures, and poor documents influence statistics significantly.
  • Number of documents: a training set and a test set should have enough documents to train the model well on various templates.
  • Rare fields: required fields with low representation (appearing <25% of times) pose a challenge to STP.
  • Extraction complexity: line items, multivalue fields, and fields with complex postprocessing or grouping can be a risk for statistics.
  • Validation rules: business rules applied to reconcile or validate extracted values should be straightforward; otherwise, there will be a low STP rate.

Success criteria help to keep the focus on the project. The project goal sets a target or an intended outcome, while success criteria help to understand that this outcome has been achieved. Metrics are used to track the movement toward a goal. For example, you can set a goal to reduce costs and process more clients with the same team. Success criteria can be a quicker document turnover identified by 40% reduced time for a claim processing, and 15% documents is STP.