Edit

Before ML project start

The section describes the peculiarities of implementing an ML project, including prerequisites and typical SOW of ML project, life-cycle, measuring results, conducting final test and dealing with the issue of low stats.

Basics

As a preparation for starting on an ML project, familiarize yourself with the basic information:

• ML AutoML glossary that explains the terms related to AutoML components in WorkFusion
• ML project questionnaire that lists all the questions that should be answered for ML project to adequately estimate and plan it
• AutoML architecture, including general AutoML components communication scheme, components communication while training model, components communication while extraction using model, and AutoML installation types (Standalone vs Cluster) along with explanation of architecture components
• ML Methodology is a collection of articles on the topic of best practices in managing ML use case. Some of the most relevant for DM topics include DA and SME effort calculation, Use Case lifecycle, ML Use Case Prerequisites. ML Methodology will be referred to further in this section where it is applicable
• System requirements and how they affect the time of training (how many models/fields can be trained in parallel)

Prerequisites of delivering ML project:

• ML (and in majority of cases also OCR) server(s) installed according to the requirements and based on required capacity or scaling; ML health check run
• Scope is clear and validated

Imagine you were given typical project where you need to:

• Classify four types of documents: Type 1, Type 2, Type 3 and Type 4
• Extract information from Type 3

Let's assume you're clear about what fields to extract, for classification though couple of hints you may use to be sure scope is defined well:

• Are those four types represent whole production throughput?

• If not, what are the other classes and how many of them in total?

• Are there any aggregate type of classes, like "Other" or "General"?

• What is class distribution? You should focus on those covering most of the throughput.

• How similar are the classes? For example, if you consider color criteria: while it's easy to distinguish between black or white, it can be much harder to do so against jeans color.

• Make sure you know how multi-class classification works at Machine Learning Basics.

• Customer aware of the need and ready to provide:

  • Documents grouped by structures.
  • SMEs for tagging of the training set in sufficient number.

  The number of documents to be tagged will likely be mentioned in SOW or needs to be determined with the help of MLE and based of what customer has available. It is also better to ask the customer to group the documents by unique structures to make preparation for tagging and OCR tuning quicker. If the documents are not grouped by customer, it will have to be done by DA. In this case DM should take into account that such grouping requires additional time. To request sufficient number of SMEs, you need to go by how many documents need to be tagged, how long it takes to tag one documents and what timeframe can be allocated for tagging in your project plan. To calculate the number or SMEs you should request use DA and SME effort calculator. It might also be the case that customer can provide only specific number of SMEs - for example, one - and you need to plan the tagging timeframe and start of model training iterations based on these resources availability. You also might find that the needed resources are available only during specific time period - for example, due to vacation, being busy due to end of financial quarter or month closure, etc. Make sure you communicate all the resources requirements and the impact of lack of resources/delay on the project plan and end date. Providing sufficient resources in timely manner is customer responsibility that is defined in SOW.

  note

  Though DAs can participate in tagging where it necessary or beneficial for the project, tagging of the dataset is usually delegated to customer's SMEs for a couple of reasons:

  • Data Analyst does not always have the time to tag training set - they need to be dedicated to other tasks described below.
  • Data Analysts do not have the expert knowledge of the data specific to the customer and use case that SMEs have.

  When DAs take part in tagging, they still need the help and advice of SMEs from customer side. When making a decision to involve DA in tagging make sure it is not preventing them from performing their main tasks and they are not becoming a bottleneck - for example, in checking the work of SMEs.

• Success criteria for ML model specified in SOW (numeric criteria usually, automation rate + accuracy in percentage). See example below.

• Team: MLE + DA(s)

  • Machine Leaning Engineer (MLE): MLE is a specialist who possesses technical knowledge and corresponding certification to run model training, implement post-processing, as well as fix OCR errors and help DA with statistics. MLE also is responsible for creating an automatically tagged training set when it is applicable in the project. Oftentimes on smaller projects the same person combines roles of RPA developer and MLE, both training the model and building the rest of the business process around it. For more details review MLE role and responsibilities.

  If you have several models that need to be implemented in parallel, you will likely need several MLEs.

  • Data Analyst (DA): Data Analyst is a specialist who analyzes the ML use case (documents and extraction logic) and prepares the data set and calculates the statistics. Managing the data set is the overriding task of the DA that includes documents templates analysis, manual task creation; communication with and qualification of SMEs, checking their work; maintaining the highest possible quality of the data set at all time. The model can be only as good as the training set is. DA can also help MLE with the logic for post-processing as DA generally has the deepest knowledge of the use case while MLE possesses technical skills to actually run model training and code the post-processing.

  Depending on the number of documents that will be tagged, the number of SMEs provided and the timeframe, you might need several DAs. Use DA and SME effort calculator to determine the number of DAs needed dependent on the factors specific to your use case.

SOW of ML project

Before an ML project starts, customer should be already educated on some peculiarities like resources needed in it, applicable deliverables and success criteria. Such agreements will be documented in SOW and are the first thing you should review before starting ML project. Pay attention to the following:

• Success criteria:
  • Should be expressed in concrete numbers (minimal acceptable percentage), usually specified for automation rate and accuracy. Make sure you know these and other automation quality metrics, what they mean and how they are calculated. There are no recommended or universal success criteria; all depends on specific model and use case. Typically criteria around 90% accuracy and 50-60% automation rate can be encountered, at least in POCs/Pilots.
  • How they are calculated:
    • Typically success criteria is expressed in terms of average rate on document level. By "document level" here we mean the average across fields and document types. DAs and MLEs look at stats at more detailed level - for specific field or even specific document - to analyze and understand results. However, for customers it usually does not make sense to go to field level, as their main point of interest is where business value is derived - for example, FTE savings and overall effort reduction of the process. It is also disadvantageous to measure result on field level as it is highly unlikely that the model will have close enough stats on all fields - some might be higher, some lower than the desired average.
  • Fields with low representation (appearing <25% of times) are excluded
  • OCR errors excluded from stats. Minimal requirement to dpi of image documents (300 dpi).
• Number of documents in the training set and test set.
• Assurance that test set is equally distributed as a training set.
• Handwriting is typically not digitized and processed.
• Assumptions:
  • Client will provide sufficient SME (Subject Matter Expert) resources to perform tagging of documents.
  • Client will provide all training data in at the start of the project.
  • Data set is representative, based on historic or current production data (rather than a biased sample of convenient documents).
• Customer responsibilities. Again, make SMEs available to create a training set.

SOW for ML example 1

4.7 Success Criteria
4.7.1 For fields appearing in documents more than 25% of the time:
4.7.1.1 On average, more than 90% accuracy, excluding errors due to OCR conversion (for PDF documents) into XML format that occur from low quality source documents
4.7.1.2 On average, more than 50% per field automation rate
4.7.2 Execution of robotic bot(s) for non-exceptional scenarios where the inputs are valid and the business applications are not triggering any processing errors
4.7.3 Manual tasks permitting to add missed or correct wrong machine extracted data for achieving 100% completeness and correctness of processing term sheets and then verifying the data with rules-based reconciliation as specified by the customer.
4.8 Assumptions
4.8.1 WorkFusion and its subcontractor personnel will have access to all necessary systems and data to implement the process before day one of the project.
4.8.2 Tagging and processing of 1,000 email bodies containing exhaustive term sheet information and 1,000 PDF machine-typed term sheets (400x2 for training + 200 for test in both cases). Every field that needs to be extracted features 1,000 times in each case and if not, supplementary examples will be provided.
4.8.3 Sample is based on historic / current production data (rather than a biased sample of convenient term sheets that isn't representative). ...

5. Customer Responsibilities: As part of this SOW, Customer will commit to the following responsibilities in order for WorkFusion to provide its services. ...
   5.3 Make analysts available to create training set (tagging of both 1,000 PDFs and 1,000 emails) and then review data extracted by the cognitive bot.

SOW for ML example 2

1.1 Cumulative Success Criteria
1.1.1 Test 1 – Document categorization
1.1.1.1 Automation Rate: average 40% automation across document types, calculated on a test batch of 100 documents, as per results and after manual review of each document categorization within WorkFusion's Workspace. Test batch can be provided as the existing result of current categorization process and needs to be equally distributed - as the training set - in terms of formats. 1.1.1.2 Accuracy Rate: average 90% across document types, calculated on a test batch of 100 documents, as per results and after manual review of each document categorization within WorkFusion's Workspace. Test batch can be provided as the existing result of current categorization process and needs to be equally distributed - as the training set - in terms of formats. This rate excludes errors due to OCR conversion of PDF documents into XML format occurring from low quality source documents (below 300 dots per inches definition). ...
1.1.2 Test 2 – Data extraction
1.1.2.1 Test to be performed on the batch of 100 documents. The test batch needs to be well represented within the training set in terms of formats, representation and document quality.
1.1.2.2 Automation: 40% average rate at document level, evaluated based on the manual review of documents within WorkFusion's Workspace.
1.1.2.3 Accuracy: 95% average rate at document level, evaluated based on the manual review of each document within WorkFusion's Workspace, excluding errors due to OCR conversion of PDF documents into XML format occurring from low quality source documents (below 300 dots per inches definition). ... 1.3 Assumptions
1.3.1 Client will provide sufficient SME (Subject Matter Expert) resources to perform tagging of documents in scope for generating the training sets for Machine Training. 1.3.2 Client will dedicate SME resources for validating Business Process implementation in User Acceptance Testing. 1.3.3 Client will provide valid inputs for RPA automation development and sufficient number of test cases in advance of development process.
1.3.4 Client will provide all training data in the first weeks of the project. 1.4 Out-of-Scope
1.4.1 Integration with systems other than the ones mentioned in this SOW.
1.4.2 Extraction of data fields other than the ones mentioned in this SOW. 1.4.3 Handwritten input will not be digitized and processed by WorkFusion SPA's OCR and cognitive bots.