Edit

DA and MLE step-by-step guide

Introduction

WorkFusion Intelligent Automation Cloud puts in place Human-in-the-Loop (HITL) technique meaning that machine and human works together to achieve the best results in documents processing.

This concept is crucial and results in the requirement that machine and Human should produce the same outputs. In terms of WorkFusion, Model's output should be identical to the output of Manual Task submitted by experts from the customer's side or subject matter experts (SMEs).

SMEs use WorkFusion Workspace application to extract values from documents (or run any other actions on documents) via different Manual Tasks (MT). It's possible to think of Manual Tasks as buckets (for tech folks - queues), where running Business Processes (BP) put documents and Users pulls documents from them by opening appropriate Manual Tasks in Workspace.

Automation Business Process used to automate a single Manual Task does the following:

1. Pre-processes a document before sending it to AutoML server (e.g.: remove additional header tags that decrease models' confidence).
2. Sends a document with the required model details to AutoML server.
3. Runs post-processing if it's necessary.
4. Checks the response and if it's failed forwards document to a Manual Task.

If no periodically updated external reference data is used for post-processing, it is strongly recommended to implement it inside a model, not to break the concept described above and results into:

• Incorrect model's statistics (as a model produces not ideal values and there is no way to correct them).
• When post-processing is implemented as a separate step, a developer operates only with extracted values without its context (for example, a model extracted "Tom Sayer" company name. It looks like a correct value, but there is a requirement to have a company type abbreviation as part of the value if it exists. "Ltd." string exists in the document and if you have access to the context - it's rather easy to analyze it and form the correct value concatenating the extracted value and found "Ltd." one. But in case of a separate step, there is no access to the context, so it's impossible to form the correct value).
• MLE has to switch between different pieces of solution (model's and step's sources) to make automation part (model + PP step) produce good results.

ML implementation phases

Each step in the flow significantly influences the result and an output of one step drastically influences complexity of the next ones. So, if the initial quality of documents is poor, it will be difficult to receive good OCR output and virtually impossible to have accurate training set that will lead to building very unstable model producing low results.

While working on ML use cases, it's important to focus on "happy path" scope: the goal should be to implement a stable solution for the most frequent documents/cases; exceptional ones can be handled by SMEs via Manual Task.

ML Use Case flow

Taking the chain of steps in the mind, lets go through ML use case flow and think about what DA and MLE folks have to do on each step:

Initial analysis of documents by DA

• Evaluate the quality of documents, if OCR produces good output. If no, raise it to Delivery Manager to negotiate it with a customer.

  Potential issues:

  • Documents are dirty, some texts are not readable
  • There are lots of handwritten notes/signatures above values being extracted
  • Documents scanned with low quality that leads to problems of reading texts

  Potential solutions:

  • Customer improves quality of scanning (increase DPIs, replace scanners, and so on).
  • Use ImageMagick utility to clean documents from "noisy" signs, for example, remove shadows, decrease size of hand-written notes, and so on.
  • Apply symbols' patterns to improve recognition of some letters or numbers (usually when non-standard font is used in documents).

• Collect requirements for each field. DA can use the following template for it. This information is used by DA and MLE to build Manual Task and Model producing the same output (e.g.: all dates should be formatted in a similar manner) This activity requires involvement of the most experienced SMEs as any errors the team find during the tagging will require reconfiguration and retagging of a data set.

  Potential issues:

  • Experienced SMEs are busy and can't join the team.
  • It's impossible to collect all requirements due to huge number of exceptions (not possible to detect "happy path" set of documents or values).
  • Customer tries to start tagging immediately saying that SMEs will have the time only now.

  All these issues should be immediately raised to Delivery Manager and he has to negotiate it with the customer. Skipping this step can lead to increase of use case complexity and struggling with model's retraining.

  Delivery Manager has to educate a customer about importance of this step and explain the risks team can face with on the next phases of the project (retagging of documents, model producing incorrect results, mismatch of output formats, and so on).

Implementation of Manual Task for tagging

Manual Task for tagging should be implemented based on the requirements collected by DA with the help of SMEs.

The following parts of Manual Task have to be coded:

• Reading and transforming values (e.g.: there is a requirement that dates have to be output in mm/dd/yyyy format. So, if there is a value "September 1, 1999" in a document, it should be correctly parsed and transformed into the correct output one "9/1/1999").
• Validation rules (e.g.: there is a "Total" field and a grouped set of products' prices and there is a rule that sum of prices is a total sum. So, an appropriate validation rule has to be implemented).
• Validation on reference data (e.g.: based on the collected requirements, there is a limited list of currencies that are in scope. An appropriate validation rule has to be implemented to validate an extracted value against reference data).
• Normalisation of values (e.g.: documents contain country names in different notation "Germany", "GERMANY". Transformation has to be coded to normalise extracted values).

Training and qualification of SMEs

When Manual Task for tagging is ready, DA starts educating SMEs on tagging rules and run qualification training on it. Description: here

note

Tagging of training set can be started only after SMEs can tag a selected batch correctly.

Tagging

The most critical and risky phase of an ML use case:

• Time-consuming.
• Depends on customer's folks.
• High-cost of post-fixing.

When SMEs tag training set, DA validates tagged documents on the following:

• Accuracy and consistency of tagging. If DA notices any issue, (s)he immediately raises it to Delivery Manager and SMEs team
• Consistency with the collected requirements. It's possible that new ones are found during tagging phase. DA has to collect them and discuss the influence with Delivery Manager and SMEs.
• Distribution of fields' values across training set. Should be done on daily basis to raise a risk as soon as it comes to life. For example, SMEs tagged 300 documents and DA sees that values for 2 fields exist only in 10 documents. So, it's clear that even if SMEs tag 1K documents, the team still won't have enough samples to train a stable model.

Delivery Manager has to raise it to a customer:

• These fields can be exceptional ones and a customer agrees to exclude them from the scope.
• These fields are still required and a customer has to collect more documents containing these fields

Model implementation

Post-processing should implement the same logic that was coded for Manual Task. It guarantees that values produced by these two artefacts will be the same.

Then, the model is tuned.