10.0

Edit

Enable Analytics

The following figure is an example of a typical OOTB AutoML Business Process (BP).

Analytics in an OOTB AutoML BP allows to collect and rate model predictions and AutoML performance against Gold (human) answers and human performance.

If Analytics is enabled, you have two options:

  1. Input is raw data without the answers. In this case, the document goes through STP or Exception pipeline, and you get the real-time Analytics.

  2. Input is a tagged document containing answers for a given manual task, i. e. the gold document for current manual task. In this case processing happens via Evaluation mode. All human work is omitted, because answers are already present. In the Evaluation mode, model answers are compared with gold answers on the Analytics step.

Enabling Analytics

To enable storing data for further analysis, follow these steps:

  1. In the Business Processes section, click on a BP to enable Analytics.
  2. On the Workflow tab, double-click Work router.
  3. In the Analytics section, select Enabled, and then click Save.

This setting is applied to each AutoML BP which is created for a given use case.

Use Cases

The following section contains common use cases with enabled Analytics.

Compare Model Predictions with Gold Answers

note

Model prediction is an answer from the AutoML service with applied Post Processing in AutoML BP.

To rate model execution using the prepared Gold answers, for example, from the initial training set, follow the steps described for your use case:

  1. Information Extraction
  2. Classification

Information Extraction

note

Gold data for an Information Extraction model is tagged text — a document produced after tagging a Manual Task in Workspace.

  1. On the Workflow tab, locate your Manual Task, and then double-click it.
  2. On the Design tab, click one of your answers, and then find the Unique Code, as in the following example.
  3. On the Data tab, click Upload Data to add your tagged text to AutoML BP. When uploading input data for your AutoML BP on the Data tab, two cases are possible:
    1. Your tagged text is placed under a column with the same name as the Answer Unique Code. Mapping is performed automatically, no extra actions required.
    2. Your tagged text is placed under a column with a different name. In this case, click Map Columns and manually map that column to the Answer Unique Code.
note

If input data has other column with the same name as the IE answer code, you should rename it manually. Otherwise said column will be processed with auto-mapping instead of the desired column.

  1. Run your BP. Click the Run tab > Actions > Run.

All records after the Machine learning sub-process flow to the Evaluation mode branch directly to the Analytics step. See the following section for details.

Tagged text can be provided as a link to tagged documents.

Classification

note

Gold data for a Classification model is expected classified class.

  1. On the Workflow tab, locate the Manual Task, and then double-click it.

  2. On the Design tab, click Add Answer, and then find the Unique Code, as in the following example.

    In the example above news is a Unique Code for classified data. Note, that the answer type for the Classification should be Information Extraction.

    Here company_name is a Unique Code for classification decision.

  3. When uploading input data for your AutoML BP on the Data tab, make sure your data meets the following requirements:

    1. Classification data: one column contains classified data (original text) and has the same column name as the classified data Unique Code.
    2. Classification decision: another column contains gold data (classification decision) and has the same column name as the classification answer Unique Code.
caution

There is no automatic column mapping for the classification Unique Code. Automatic mapping is still available for the classified data column.

  1. Run your BP.

All records after the Machine learning sub-process flow to the Evaluation mode branch directly to the Analytics step. See the following section for details.

If you use simple model execution, provide only the column with classified data in input data. See the following section for details.

Model predictions in daily model execution

When Analytics is enabled, AutoML BP collects comparison information for model predictions against human answers.

  • If the processed document goes through the Exception: bot assisted or Exception: fully manual branch, then the model answer is compared against human answers.
  • If the processed document goes through the Straight through processing (STP) branch, then the model answer is compared against itself.
  • If AutoQC is used and the document goes through the Manual Task branch, then the model answer is compared against human answers.

Collected Data Usage

Comparison data is saved in a data store with name according to the following pattern: automation_stat_[bp_campaign_uuid].

bp_campaign_uuid equals to _sys_bp_campaign_uuid field after the Machine Learning sub-BP with the following replacement: "[^\w]" with "_" (underscore).

The data store has the following structure by the Column name.

run_uuid

Current BP instance UUID.

  • Data type: TEXT

field_name

Answer code.

  • Data type: TEXT
  • Aggregation: per field in document.

group_name

Group name if field_name relates to a group.

  • Data type: TEXT
  • Aggregation: per field in document.

gold_value

Gold answer, if any.

  • Data type: TEXT
  • Aggregation: per field in document.

extracted_value

Model answer, if any.

  • Data type: TEXT
  • Aggregation: per field in document.

TP

Metric for a given answer code: True Positive.

  • Data type: INTEGER
  • Aggregation: per field in document.
  • Additional information: used INTEGER because BOOLEAN is not supported by Data Store plugins.

TN

Metric for a given answer code: True Negative.

  • Data type: INTEGER
  • Aggregation: per field in document.
  • Additional information: used INTEGER because BOOLEAN is not supported by Data Store plugins.

FP

Metric for a given answer code: False Positive.

  • Data type: INTEGER
  • Aggregation: per field in document.
  • Additional information: used INTEGER because BOOLEAN is not supported by Data Store plugins.

FN

Metric for a given answer code: False Negative.

  • Data type: INTEGER
  • Aggregation: per field in document.
  • Additional information: used INTEGER because BOOLEAN is not supported by Data Store plugins.

score

Model answer metric: score for a given answer code.

  • Data type: TEXT
  • Aggregation: per field in document.
  • Additional information: used TEXT because DECIMAL is not supported by Data Store plugins.

confidence

Model answer metric: estimated Accuracy for a given answer code.

  • Data type: TEXT
  • Aggregation: per field in document.
  • Additional information: used TEXT because DECIMAL is not supported by Data Store plugins.

document_id

Equals to hitUUID.

  • Data type: TEXT
  • Aggregation: per document

doc_html_link_tagged

Empty by default.

  • Data type: TEXT
  • Aggregation: per document

timestamp

Record insert timestamp.

  • Data type: TIMESTAMP
  • Aggregation: per document

environment

WorkSpace environment type: production.

  • Data type: TEXT
  • Aggregation: per document

ml_process_time

Process time of the automation-extract plugin (post to ML service + ML extract + etc). In milliseconds.

  • Data type: INTEGER
  • Aggregation: per document

human_process_time

Human spent time. In case of multiple workers, total time spent. In milliseconds.

  • Data type: INTEGER
  • Aggregation: per document

human_process_type

Type of human work: FULLY_MANUAL, BOT_ASSISTED, EXPIRED.

  • FULLY_MANUAL — manual task without ML answer assistance.
  • BOT_ASSISTED — manual task with ML answer assistance.
  • EXPIRED — manual task is not processed and overdue.

Manual step in AutoQC is not accounted for at the Analytics stage.

  • Data type: TEXT
  • Aggregation: per document

Collected data is used to build Cognitive Automation Breakdown chart.

Comparison rules for non-empty gold and model answers

Use the following comparison algorithm to calculate statistical metrics (TP, TN, FP, FN):

  1. Split gold and model answer with '|' (pipe).
  2. Sort the resulting arrays.
  3. Compare arrays by elements with the same position.

This comparison method is used with multi-value fields saved in one string with a pipe separator. If the field is not multi-value and still contains a pipe '|' then such comparison is the same as using equals.

← Run AutoML Business ProcessView Business Process results →