ML Project Results
Measuring results
Common customer questions
What are model results calculated on? Were these documents used for training?
ML model results are represented by statistics calculated on a test set of documents that are representative of the dataset and have not been used for model training. The statistics are calculated by DA and compiled into a report.
It is beneficial to present final results of implemented ML use case in light of the impact on client business. Most often that would be potential savings in terms of decreased number of employees (FTE) that have to be involved in the process after use case is deployed to production. It is recommended to gather metrics reflecting how much time/money is spent by client on running the process before automation is deployed and compare them to the metrics after automation is in place. In some cases overall process accuracy and predictability could also be improved after implementing automation. Try to analyze this aspect too and present it to client along with the ML model stats.
Conduct final test
A final test should be agreed in detail with the client in advance, clarifying the following aspects:
- How the statistics are calculated. Usually the same formulas are used by WF unless customer specifies otherwise. However, you must clarify in advance if the customer expects the statistics to be calculated in a different way. First of all, know your SOW - the way of calculating statistics may be described there. Oftentimes SOW includes lines stating that OCR errors and fields with low representation are excluded from stats calculation.
- Test set creation:
- Size of the test set (usually ~20% of the training set size).
- Provisioning SMEs to tag the test set in advance to calculate/adjust model stats.
- Who chooses documents for the test set. Customer might prefer to choose the documents - in this case you must have your DA participate in choosing the documents and review the test set for the purpose of making sure it is representative. Otherwise the final stats will be skewed.
- Way of presenting the results. Some of the ways (oftentimes combined) can be:
- Showing the average document level stats in a deck.
- Table with field level stats.
- Launching the test set documents tagged by the model in a human task for SMEs to review manually.
- Send a report of extracted data to customer so they can also calculate the stats from their side.
Low stats and what to do if it happens to you
In this section we will discuss the common reason that can lead to the model statistics being below the desired level, as well as the ways to overcome or manage them.
The below problems can be identified either at the start or in the middle of the implementation or at the very end close to presenting the final results. In the first case the preferred approach would be to try and adjust the plan, take corrective actions proposed here or identified by the implementation team. In the second case or due to other reasons - such as lack of time or resources to take corrective action - you might have to follow a different approach that would mostly focus on communicating poor results in the right and positive way. When you have to take this approach, it is key to be aware of the reason for the results you are getting to be able to explain them in constructive way and propose action plan that can be followed in the next phase or future project to build on the current achievements and improve them to desired level. Lack of understanding of the way ML works by the customer can lead to suspicion and negativity that is overcome with open and clear guidance through what was done, what it resulted in and why, as well as what to do next. Start with Machine Learning Basics to be prepared to speak on the topic of Machine Learning.
Dataset quality Though customers find it hard to believe, dataset quality is the most common reason of low model stats that WF come across in deliveries. Get acquainted with the dataset requirements to know what a quality dataset for model training is and be prepared to understand the pitfalls found by DA as well as discuss with customer the reasons of the low stats and ways to improve the results. "Garbage in - garbage out" is the main principle of supervised machine learning, which means if the training set is not good enough and big enough, the model won't show good results. In other words, if the accuracy of you training set is, say, 87%, there is no way the model output will go above 87% and reach the desired 90%.
- Quality of tagging
Common customer questions
What do you mean by "incorrect tagging"? People who tagged the dataset are the experts, shouldn't they know better? Why not all of the documents tagged by SMEs were used in model training?
Most commonly it is the quality of tagging done by SMEs that undermines the dataset quality. It is also the most difficult reason for customer to accept, as SMEs are the experts who know best how and where to find the needed data so it seems they by default cannot be wrong. And to a large extent it is true - the problem is that ML models perceive data different from how human do it, with a large focus on data context, not simply content. Therefore, for SME it does not matter where to take the data from as long as the data is right - such as take customer name from the header of the invoice in one record and from the signature in the next record when the same name is available in both places. For the model such inconsistent tagging poses a huge problem, undermining model confidence and therefore lowering the results. For that reason it is so important to have a DA teach and qualify the SMEs, check quality of their tagging and correct them in timely manner, and use other tools for boosting tagging accuracy like Adjudication or Moderation tasks. Additionally, you can still find that SMEs and even DAs make mistakes just due to the nature of human. On average human accuracy is estimated at 90-95%, so reviewing the dataset is an always actual activity that can help to achieve better results without doing any other changes to the model that require developer involvement.
- Representativeness of the training set and test set
Common customer questions
Why the model did not extract well from this document? Why is it necessary to involve DA in choosing documents for the (final) test set?
Another driver of dataset quality is its diversity. One of the reasons of low stats may be that some document template is not represented or not enough represented in the dataset. To understand if that is the case, check the documents that have lowest stats after model extraction (records with highest number of false negatives and false positives). If you identify that there are very few or no documents with similar layout in the dataset compared to other layouts, it is quite understandable why the model has not learnt how to confidently extract data from them. A solution could be to add more documents of such template if the customer has them available or explain that the extraction can improve after more of such documents are accumulated in the normal course of work. Similar situation can occur in the calculation of the final stats on a test set, if the test set was not properly prepared to be representative of the population of documents in the training set. In such case the test set might contain template(s) that were not included in the training set, which means the model was not trained on them at all and is unlikely to extract data from them well. In this case the most logical solution would be to exclude such templates from the test set or at least explain how they skew the data and why. The option of reshuffling the training and test set and retraining the model to include the missing template is unlikely if we are dealing with the end of a POC, but might still be an option if it is a production project and the time allows to retrain the model.
OCR quality. Low model statistics could be caused by the issues with the OCR. In case of issues with the document quality or with OCR project team might need to do retagging, model retraining and do changes to post processing implementation. Hence it is a must for every project to plan activities related to verifying OCR output and making sure all necessary values are there. See more details in the below section related to OCR.
Representativeness of fields
Common customer questions
Why this field cannot be included in model training? Why the model fails to extract this field properly?
Check the field-level stats to see if some specific fields are pulling down the average figure. If so, one of the reasons it is happening might be that these fields appear less than others in the dataset. Check the number of these fields available in gold data (such stats can be checked in WF UI when automation training set is chosen in automation settings of a manual step). If these fields are hardly present in the dataset, it means the model would not be able to learn from them properly. In such case, similarly to the case when some document templates are not represented well enough, adding more samples with such fields to the training set should help. If such agreement was made with customer, these fields can be excluded from model stats calculation until their number in the dataset reaches desired level (for example, 25%). In some situations the fields that were initially included to the list for automation in fact appear so rarely that no or hardly any samples for them can be found throughout the project. In such case it makes sense to exclude them from the training set at the very start. They can be still left in a manual task for the users to be able to extract manually when they encounter such fields in the course of work, but should not be tagged for the training set
Difficult fields
Common customer questions
Why the model fails to extract this field properly? Why do SMEs need to change the tagged text's value to a different format (normalize)? Why are the stats so low in the first model training?
Another reason why some fields show lower stats than others might be that such fields are more complicated, for example, free text fields, address, requester name. Then you can focus on improving the average stats through accuracy and automation rate of simpler fields and adjustments to the model that would help to better recognize the difficult ones:
- Improve post-processing. Identify if there are any validations and rules, for example, that can be derived from other related fields - that will help to make better decisions on the complicated field. MLE and DA can do this with the help of SMEs.
- Custom model required. In the worst case you might find that the field is so unusual and complex that generic model just cannot handle it well enough. Usually, complexity of fields will be estimated prior to project start, but such estimation cannot be exact until you actually try to train the model. Consulting with ML COE will help to understand if there are any ways to solve the problem through postprocessing or further development would require involvement of Data Scientist.
Consistency in calculation
Common customer questions
Why are the stats lower on the model trained with more documents? Shouldn't the quality grow with addition of more docs?
This point is more related to model learning curve. If you expected to see model stats improve over time but cannot identify a pattern in statistics, one reason for it might be inconsistency in the way that stats were calculated. If you compare stats among model training done on different number of documents in the training set, the model version, postprocessing applied, and test set you are measuring on should all be the same.
Another typical case where you can come across this problem is if you use the whole training set instead of a test set for calculating and comparing stats. Then with adding more documents in the next iteration you will have different number of documents you are comparing as well as potentially different template representation in the two training sets. It is natural to observe model stats (calculated on the whole set) dropping when you add new documents - the model indeed might extract them worse, than the previous set of documents. However, in reality it does not mean that you actually got worse result - now the model might extract from the new documents at least something, while before it could not extract anything. That is why it is so important to have a proper test set - in the described case it should be comprised partially of the old documents from the training set, partially of the new documents added to it recently. Both models should be compared to this same test set and then you would be able to see the real change in result.
Natural reasons
- Dataset saturation / flattening of learning curve
To add more to the topic of learning curve, the model quality grows with addition of more documents only if new documents are adding more representation to the training set. If all of the variations of document templates are already present in the dataset in enough quantities for the model to learn them, then adding more documents of the same layouts will not increase the quality any longer. What you will observe is that the model learning curve flattens at some point, and depending on how complicated the documents are and how many variation they contain, the faster it will happen. For example, if there were some 5 different templates, the documents have repeated simple structure, and you managed to collect an equally distributed training set containing all of these templates, you might have the model learn all it could already on ~300 documents and stats not growing further with adding more records. The only way to grow the quality and automation rate in such case would be to improve postprocessing or ultimately customize the model. In cases when the dataset is more complex, thousands of documents will be needed to achieve comparable results.
Below is a representation of a typical learning curve of the model - steep at the beginning and flattening at some number of documents added to the training set / training iterations passed. Do not read it as prediction. This is a fake depiction only.
*- Model quality limit
Last but not the least law to remember and always communicate to customer is that model quality will never be 100%.