Analyze results
Along with the out-of-the-box operational analytics of the WorkFusion’s platform, the Adverse Media Monitoring AI Agent includes pre-built dashboards to enable real-time, ongoing monitoring of the solution for operations managers. The dashboards can be dynamically filtered to visualize data from a specific Business Process run, timeframe, or parameters, such as risk category or news domain.
Adverse Media Monitoring analytics is presented with the Overview and Articles Analysis dashboards providing sophisticated views of solution effectiveness. They allow you to track overall performance, search for improvement opportunities, and identify and troubleshoot issues.
To open the Analytics dashboards, in the Control Tower menu, click Analytics. Here, you find the General group for the Overview dashboard, and the Custom group for the Articles Analysis one.
Overview dashboard
The Overview dashboard focuses on processing time and solution quality. It allows you to track the overall manual processing time, volumes of search requests, processing time by Business Process step, model accuracy visualized as a confusion matrix, and list of keywords used by the model to score each article and categorize by risk.
Filtering options allow you to select individual Business Process instances, specific process runs, and search periods.
The dashboard includes the following sections:
Manual Task Processing Time
The area displays the average manual handling time per entity reviewed and the volume of tasks completed over time. Here, you can check how Average Manual Processing Time vs. Number of Tasks submitted on a specific day change over time.
It also displays the summary information on the Average Number of Tasks submitted daily and Average Manual Time for a selected period.
Example: over the last seven days, the Average Number of Tasks was equal to 401 per day with an Average Manual Time of 1 minute 23 seconds per task. You can observe stable behavior on the manual processing step with no dependencies on the volume of tasks; hence the team works well.
Searches
The total number of searches performed daily with a breakdown by the investigation status allows us to evaluate and understand the volume and trends of investigations to be performed daily and its relative volumes in all searches.
Search Processing Time
The section provides information on Average Processing Time broken down by individual steps in the Business Process and dates used to identify bottlenecks.
Here, the process step comes from the Business Process design and corresponds to the building blocks and their names. Thus, it allows you to identify issues within processing time, when they happen, and specific steps in the process that may require attention and further troubleshooting.
Note: Cases without articles that lead to manual review steps are filtered from this view. For example: on Day 3, the times for processing steps slightly increased. That may indicate an increased number of searches, or that the system was slowed down by external factors.
Confusion Matrix
This section evaluates the solution's accuracy by comparing the model’s decisions against analysts’ decisions on whether real adverse media was found. Red boxes indicate instances in which WorkFusion’s decision differed from the analysts’ ones.
Model results are estimated based on High-Risk articles found during the search according to the logic below:
True Positive: at least one High-Risk article was found and a human operator decides to proceed with the investigation. The model results correlate to the human decision triggering an investigation.
False Positive: at least one High-Risk article was found, but a human operator decides that no investigation is required. The model incorrectly assigns high risk, and more time is spent on review.
False Negative: no High-Risk articles are found, but a human decision is to proceed with an investigation. The model did not identify any risk that leads to missing risk.
True Negative: no High-Risk articles are found, and a human decides that no investigation is required. The model results correlate to the human decision with no further study required.
Pay special attention to the False Negative category that may lead to missing risks. The model could not identify high risk in search results, and only manual review allowed handling the case properly. The False Positive category should also be monitored as it indicates High Risk and may lead to increased manual time, while the organization's risk is minimal.
Note: When no articles were found, the alerts are filtered from the confusion matrix as no manual review was performed.
Keywords
This section contains a list of keywords and how frequently each was used in the model to score the articles and assign appropriate risk category. You can export this chart's content in the text format for further references and use it for model revision and future retraining activities.
To export the list of keywords:
Click the white space in the chart area. Then, in the toolbar located at the dashboard's right top corner, click Download.
In the pop-up dialog, select the Data or Crosstab option to get underlying data in the text format.
Articles Analysis dashboard
The Articles Analysis dashboard gives an overview of the technical aspects of the process. It provides an understanding of processed article volumes, sources of information (domains), distributions by risk category for each domain or in total, distribution of articles by size, successful parsing of articles found, and its dependency on the size of the article.
Filtering options allow to select individual Business Process instance, and specified process run, the period of search, specify news portal, the success ratio for article parsing status.
- Color coding is used consistently across all views with every single color having the same meaning within the scope of the dashboard.
- The dashboard supports inline filtering actions. That means that selecting any entity in one view will filter the other views according to the chosen value and selected filters.
Articles by Domain
The view provides information about all the domains from which articles were sourced (BBC World, New York Times). The first section shows the Total Number of Articles retrieved from a specific domain. Ther breakdown by the parsing status displays overall volumes, primary sources of information, and parsing quality of each domain. The second section on the right indicates the distribution of successfully parsed articles by risk categories: High, Medium, Low, and No Risk (percentages are calculated relatively to the Total Number of Articles).
Example: The New York Times is the primary source of articles, and it was parsed successfully in >96 % of the cases. It has an expected rate of High-Risk articles within several percent (1% here). The Hill news website has high parsing success; simultaneously, it brings a more significant contribution to High and Medium Risk articles. Here, additional training may be required to minimize reviews.
Articles by Risk Category and Parsing Status
The section shows the distribution of articles by risk category to assess levels of risk over time. It provides a review of the total number of articles that were not parsed and how they are distributed among risk categories: High Risk, Medium Risk, Low Risk, or No Risk.
Parsing Success Based on Article Size
The chart provides information on the articles' distribution by content size and error status during the parsing step to understand the size of commonly processed articles. This view also allows you to evaluate how effectively the parser works with various article sizes and identify areas for improvements. The section is most relevant when using Google News or Google.com and its local variants, such as the news provider rather than a curated news provider (for example, Factiva).
Each bar represents a bin for article size (word count), and it can be changed using the Article size increment parameter. The first bar has the zero (0) label and contains all articles within the defined size increment. See the example and description below.
Note: Article size increment is set to 1,000 words, and the first bar (zero/0) represents all the articles with the size from 0 to 1,000 words.
