Edit

Use Case lifecycle

The process, with phases described below, has been proven to work and practiced by WorkFusion PoC while delivering complex ML+RPA implementations for customers.

On a high level, each Use Case goes via five main phases. Each phase has its requirements, steps and roles involved.

Implementation phase stages

Let's logically structure a typical Use Case Implementation in stages defined by separate people doing specific work and handing-off results to the next person:

An implementation consists of ML and RPA parts. RPA deals with well-structured and repeatable processes and develops automation that replaces manual tasks. ML development delivers a model that handles unstructured or semi-structured data in the documents. The model is trained on a data set collected from original documents. A high-quality data set is the Data Analyst's main responsibility. The better the quality of a data set is, the higher results of ML. Implementation can be delivered as PoC (Proof of Concept), Pilot, or Production, depending on the scope of the Use Case.

1. In the data set collection, the Data Analyst studies business logic of the Use Case and applicable documents, and defines the rules and corner cases. The Data Analyst works closely with the Subject Matter Experts (SMEs) from the customer/partner side on the Data Set collection step. They have a deep understanding of documents' logic and tag the data set for model training. The Data Analyst trains SMEs to tag the documents and verifies tagging.

2. The Machine Learning Engineer or Automation Engineer Specialist leads model training and the Data Analyst analyzes ML results and defines rules that help to improve the model. ML results are evaluated by comparing the results provided by people (manual extraction or classification that is considered to be correct) and calculating quality metrics statistics. The Data Analyst calculates interim and final statistics and submits these to the customer in the form of a report.

3. At the analysis of results stage, the Data Analyst needs to review the model execution results and, if necessary, propose ways to improve. The model is trained on the test set for each training iteration. If there are several iterations, the Data Analyst calculates statistics of each iteration based on evaluation results. The Data Analyst needs to calculate statistics for each iteration and analyzes the delta for tagging iterations and model mistakes.

4. ML improvements and retraining is handled by the Machine Learning Engineer or Machine Learning Engineer + Data Scientist. This is the most complex stage, as it includes creation of custom model components.

5. The (first) final report of automation results has to be shared with the customer, together with any reasons why some stats on some fields may fall below success criteria. Moreover, further steps and solutions should be proposed.

6. In the integration of ML results into RPA stage, structured data made by machine learning is entered into organization's internal systems. Typically, RPA is used for that because corporate applications lack API support and the only way is to have bots simulate user interactions with UIs.

7. During the final report, analytics helps DAs and business users to see and understand outcomes of automation in production. This may include aggregated numbers showing savings, amount of work done, SLAs, average ML measures, and so on.

The ML Use Case lifecycle stages are as follows:

Project stages

The following is a guide for project milestones that reflect the particular level of the use-case flow. Milestones distribution by project Sprints maybe different from given above depending on project scope, available FTEs and hardware.

Information Extraction with SME Tagging:

• Stage 1. Initial Setup, SME Qualification
  After setting up the process, DA should qualify SMEs to tag initial data.

• Stage 2. Initial Data CollectionCollect data while validating that SMEs submit quality data

• Stage 3. Model Training and Model Extraction

  During the stage, tagged data should be collected. We recommend 20-50 documents of each template of each type in a data set depending of the use case complexity. Visit our Data Analyst Manual to learn more about data set collection.

  important

  Before proceeding to the next phase with final documents, the results of Milestone 2 must be reviewed for accuracy as well as balancing expected document layouts.

  The training set composes 80% of the whole data set. Test set composes 20% of the whole data set not used for model training. The aim will be to run extraction on documents that haven't participated in training. The second aim of the test set is to check the model performance for possible exception cases in production.

• Stage 4. Model Evaluation and Model Tuning

  ML improvements are implemented and retraining is run if needed.

• Stage 5 Report

  The DA provides a report.

Classification with historic data

• Stage 1. Initial Setup

  Set up the process to take the data as input and produce the proper outputs.

• Stage 2. Initial Data Collection

  Collect data while validating that SMEs submit quality data.

• Stage 3. Model Training and Model Extraction

  Train and test on the full set:

  • Data Set should contain 100% of the data.
  • Set aside 20% of that for evaluation purposes.

• Stage 4. Model Evaluation and Model Tuning

• Stage 5. Report

Define document tagging timeline

For more precise defining document tagging stage timeline, refer to the DA and SME effort calculation.

The calculator allows to define:

• How may days are required to create the data set with given parameters.
• How many time SMEs will spend to create the data set with given parameters.
• How many time DAs will spend to verify the data set with given parameters.

Define ML training timeline (versions 9.0, 9.1)

Time depends on Search Engine time limit you have, number of fields and ML cluster capacity.

Default Search Engine time limit was changed to two hours.

Training time ~ Search engine Time limit * (number of fields / number of fields in parallel on the server). From CT 9.1 there is an option to see the estimated training time from the UI.

According to the system requirements, with 8 CPU and 32 GB, you can train up to 8 fields during 2 hours concurrently.