Analytics for Digital Workers
The goal of the article is to provide a high-level overview of pre-packaged solutions the WorkFusion platform offers. We will also review how to analyze the impact of the solution, find potential issues, and address them.
What are Digital Workers?
A Digital Worker is a ready-to-go automation solution that covers a single end-to-end Business Process and is built through best practices and experts' knowledge. The solution commonly includes configurable inputs and can be adjusted to specific customer needs, ready to use for a Business Process that automates a particular function, and with configurable output formats that can be customized and adapted to particular needs and compliance rules.
Two major Digital Worker skills are:
- Adverse Media Monitoring (Negative News Screening) from the Anti-Money Laundering process group
- Identity Verification from the Account Opening process group
Analytics for this type of process comes in two forms: general analytics to understand the process flow and look for common technical issues and troubleshooting; and specialized, as part of a pre-packaged solution, which gives a particular and business-oriented context. For this, there is a specific extension in Analytics that adds specific views and level of detail for its specific process and differs from one Digital Worker to another.
Evaluate process performance and find bottlenecks
On a high level, general automation analytics relates to business impact, overall process performance, stability, accuracy, and quality. The key metrics here are:
- Manual Handling time required to process a single business entity
- End-to-end Processing Time
- Volumes processed daily (weekly, monthly)
- Process Accuracy and Quality
- Process Bottlenecks (people or technology)
- Input Data Quality is an external factor and drives solution quality and accuracy
- Customer Satisfaction (measured indirectly and primarily by looking at End-to-End Processing Time and Accuracy)
These flows and metrics are common across various business areas and provide a full picture of how well the process works, helps to identify issues and bottlenecks — whether they are related to technology and implementation or some other cause. More specific analyses require switching to different views. In the case of pre-packaged solutions like Adverse Media Monitoring and Identity Verification, the critical evaluation will happen according to the above metrics. At the same time, business context is required to correlate the metrics to actual business needs and make accurate decisions—that is when pre-packaged analytics for Digital Workers comes into place and allows users to get a quick overview on the operations and process efficiency isolated to a particular business case, with required details and context for a business person.
Analytics for Digital Workers
As mentioned earlier, Digital Worker analytics follows the standard metrics and thinking flow, enriching the metrics with a Digital Worker context and details required to make better decisions. While there is a set of various metrics covering the whole process and platform health, one of the critical aspects, when we talk about processes like Adverse Media Monitoring and Identity Verification, is the Turnaround Time, SLA, and Customer Satisfaction. This mostly relates to Manual Handling Time when we look at the overall process, and the key for a business person is to quickly find whether operations are performing well and, if not, what may be the root cause. Below are the most common scenarios to follow working with Digital Workers.
Adverse Media Monitoring
Adverse Media Monitoring (or Negative News Screening) involves searching for negative news about a person or business. It can be a crucial part of Know Your Customer (KYC), and Anti–Money Laundering (AML) processes for banks. Negative news checks are essential for uncovering a customer's possible involvement with money laundering, fraud, organized crime, and terrorism, among other threats. Conducting Negative News Screening ensures that firms do not become unwittingly involved in criminal financial activity. Regulatory bodies enforce banks to perform screening process on companies, their top executives, and shareholders.
The Overview dashboard for Adverse Media Monitoring provides standard business metrics on the process:
- Manual Handling Time
- Search volumes
- Quality: the Confusion Matrix represents quality; additionally, it has Digital Worker-specific categorization of each task by the resolution
- List of keywords used by the model
The second Domains dashboard provides a more technical and detailed view on how the process works and what type of content being processed and handled by the analysts.
These two dashboards allow you to get a quick overview of the process, evaluate its efficiency and quality.
Keywords list provides information on words and phrases used in the solution to score articles and identify the risk category they belong to:
- High Risk
- Medium Risk
- Low Risk
- No Risk
When reviewing this list, an SME looks for words that may be missing and often used in manual work to decide on actual risk and investigation.
Recommended action: If anything is missed, re-train the model in the Adverse Media Monitoring process and include new words and phrases. This will lead to improved accuracy and will be reflected as a decreasing number of False Negative and False Positive cases in the Confusion Matrix.
Risk distribution assessment — review distribution of articles by risk categories, to identify process improvement needs:
A large number of High or Medium Risk articles leads to an increased workload for analysts, tasked to review more information and decreases the overall number of cases that can be examined in a specified time — for example, per hour.
Recommended action: If the number of such articles is expected, the process may require more analysts to be assigned to manual reviews to increase process throughput.
A large number of High or Medium Risk articles leads to an increased workload for analysts, and may indicate poor model performance.
Recommended action: Review model accuracy and list of keywords and re-train the model to improve performance.
The high number of Not Parsed articles indicates poor input data quality. It may show formatting changes in source portals. Recommended action: Update the process to handle such articles effectively.
Distribution by content size indicates two potential problem areas:
Articles of a specific size (for example, 0–500 words) are failing to parse. This means such documents are of poor quality or source pages contain many ads.
Recommended action: Analyze these articles and the source portal, identify a technical root cause why the parser fails, and update it accordingly.
High numbers of articles of large size will impact analysts' manual review time, as such articles, mainly appearing on Medium Risk bucket, often will lead to increased manual handling time.
Parsing success rates by Domains serve as overall indicators of incoming data quality and how effectively the process can handle each portal when processing the content. The root cause of high failure rates is on the technical side — a new source of information may appear, and the solution is not optimized to effectively process the content, formatting, and implementation of a portal could have been changed. In both cases, the solution needs to be adopted.
Distribution of articles by risk category for each domain allows reviewing of all sources and assess importance. For example, domains that are feeding only low-risk materials can be ignored or put into a blacklist to reduce the amount of noise; domains with an unexpectedly high number of High Risk articles may require additional review to confirm that is the case; and no technical issues or opinion bias in a specific source may lead to an incorrect decision on particular names searched.
Practical cases
When working with Adverse Media Monitoring and checking daily, you may face the following common scenarios. Below, you will find a description of how to find and address some typical situations.
Case 1: Missing keywords
Problem: Some critical words are not included in the search. This may affect decisions in assessing actual risk and need for investigation.
Description: Keywords list in Adverse Media Monitoring should be verified periodically to control overall model quality and ensure that the solution works at the highest possible accuracy. Based on analysts and subject matter experts' knowledge, a user can decide whether the model requires attention and needs to go through technical review and modification.
A user should check the Keyword widget on the Overview dashboard and verify that it's not missing any well-known phrases for high-risk articles. For example, the list does not contain "crime," "offshore," "third party," "family," or "organization," so this would lead to more frequently marking the content as "high-risk" and land it on the investigator list.
Solution: To fix this case, discuss with the technical team and provide a revised list of words to be included in the model as indicators for high risk. The model will then be re-trained and published to production.
Case 2: Increased amount of "high-risk" articles
Problem: A large number of "high-risk" articles affects the total amount of requests that can be processed daily by analysts.
Description: A high number of articles requiring review, validation, and confirmation may have a significant impact on manual work to be performed by the analysts. Ideally, you minimize the Manual Handling time and maximize the overall Throughput.
To check the current state and verify whether there are any issues, check the Adverse Media Monitoring Domains dashboard and pay attention to the Articles by Risk Category widget. It will indicate current distribution and highlight any issues; for example, the current rate may be 5% greater than expected.
A reason for a high number of articles associated with higher risk can be model threshold settings that assign risk categories according to a score defined by the model.
Solution: A potential fix for this issue is to modify the threshold, and in case of high volumes, a threshold increase — for example, from 0.8 to 0.9. That means only those articles scoring higher than 0.9 will be treated as "high-risk." To verify the change in production, monitor the solution health for a few days and ensure the change leads to expected results and does not hurt accuracy (where the model starts missing essential cases). This can be done via the Confusion Matrix.
Case 3: Increased Manual Handling Time
Problem: Manual Handling Time is increased due to lack of knowledge and skills.
Description: Manual Handling Time is a significant metric, as it directly impacts the volume of requests that can be processed daily, the process throughput, and customer satisfaction. The quicker that checks can be performed, the better. On the Adverse Media Monitoring Overview dashboard, the Manual Handling Time chart has a central place and allows you to review trends and current state.
Solution:
It is essential to look for current volumes concerning manual handling time to verify there is no correlation, and processing time changes are not a result of changed amounts.
When you see increased manual handling time, the next step is to check the Manual dashboard (an out-of-the-box dashboard) and perform analysis in the scope of the Adverse Media Monitoring Business Process. The Manual dashboard will highlight whether any of the analyst team members are performing worse than others.
When you see a high number of long processing times for one of the analysts, you should discuss it with them: Verify whether they have appropriate knowledge on the process, and run education session to help them perform better.
Account Opening
Identity Verification is the process of checking a customer's ID documents to ensure they are who they claim to be. It is a part of many methods, such as:
- Account Opening
- Client Onboarding
- Know Your Client (KYC)
- Anti–Money Laundering (AML) processes
WorkFusion accelerates the process of Identity Verification by extracting pertinent data from documents, such as passport and driver’s license, retrieving data from internal and external systems, and reconciling it all to uncover any discrepancies. The Identity Verification process is necessary due to KYC and AML regulations.
The Dashboard delivered with Identity Verification allows you to monitor and troubleshoot the underlying process, as well as evaluate its efficiency and quality through the key metrics such as Average Time to Verification, Automation Rate, Average Verifications Performed Daily and SLA. These metrics allow you to evaluate process throughput, understand high-level quality, monitor internal SLA, and predict impact on customer satisfaction.
Details on Country of Origin highlight lower-level issues with OCR when dealing with multiple languages, and Automation Rate by fields will indicate if there are any issues with specific area or type of document — for example, if templates are changing or there is a new type to deal with.
Practical cases
When working with Identity Verification and monitoring it daily, you may face some common scenarios. Below, you will find a description of how to find and address some of these typical situations with Analytics capability.
Case 1: Issues with language settings in OCR component
Problem: The processing of articles in languages that are not set in OCR violates SLA.
Description: Dealing with Identity Verification and document processing is a typical case: Various input document types originating from different regions and countries may come in different languages. To effectively process documents in different languages, the OCR component should be set accordingly and include appropriate languages.
From a process perspective, getting a document with a new unknown language will result in SLA violations since all those documents will require manual processing and validation, which will be seen in the SLA widget, highlighting in red cases outside of SLA. To double-check this hypothesis, we can check the Average Time to Verification widget. It should indicate an increase, and the trend line will indicate there is a spike in the most recent days. At the same time, Volumes will not show any significant variance, meaning the incoming requests are coming at the same rate per day.
We may additionally limit the scope to the specified day and through the Highest ID Volumes by Country widget; we would see that Germany has mostly Failed to Parse statuses for documents — this will indicate issues related to a specific country and prove the initial hypothesis that there could be issues with country-specific document processing.
Solution: To resolve this case:
- Modify the OCR setting and add the German language to the list.
- Re-run the process.
- Verify that processing in German ran successfully, and Manual Handling Time, along with SLA violations, returns to normal and expected values.
Case 2: Data extraction cannot recognize specific fields
Problem: Not all fields are recognized during data extraction.
Description: Data extraction is a crucial component of document processing. The issues at this step may result in different business impacts. This will cause increased Manual Handling Time, lower the process Throughput, and introduce compliance and customer satisfaction risks related to lower quality of work and increased time to process requests. As we can see, unexpected increases in processing time and SLA may be associated with a poor extraction quality, which will also decrease the Automation Rate. Still, due to high volumes of work, it may have gone unnoticed.
Solution: Checking the Automation Rate by Fields widget will indicate a poor quality on a specific field, and we can verify for what type of documents it does relate. Knowing the field name and type of document, we need to check the latest manual tasks and perform additional tagging to improve field recognition and data extraction processes.
Summary
In this chapter, we reviewed WorkFusion Digital Workers and what is currently available on the platform. We also examined the most common metrics to describe any process and how they were applied to Adverse Media Monitoring and Identity Verification in pre-packaged dashboards.
Through the practical cases, we looked in detail at how a business user may identify whether the process performs as expected or if there are areas for improvement, plus what can be the next steps and action items to improve business outcomes from the deployed solution.