10.2

Edit

Perform Automatic Quality Control

Automatic Quality Control (AutoQC) refers to the use of statistical methods in the monitoring and maintaining of the quality of products and services. In WorkFusion Automation workflow, the AutoQC sub-process chooses the optimally cost-effective combination of automated machines and cloud worker that always deliver at or above the acceptable quality level.

The main concept of AutoQC is to take a Sample from a defined Batch of items (BP records, documents, etc.) and verify each item in that sample. After the sample verification, the whole batch is considered as accepted or rejected depending on the Rejection Limit.

Therefore, AutoQC allows to verify only a part of batch, not 100% but maintain continuous quality check.

Theoretical bases and real-world usage

Detecting sample size and rejection limit

WorkFusion AutoQC is based on the Military Standard 105E (Inspection Level II) which defines the dependence of sample size and rejection limit from batch size, AQL, and inspection type.

As an input, we have the following parameters:

The AutoCQ application uses the following table as a reference to determine the Sample Size and Rejection Limit.

Extending and expiring batches

The following rules are applied to batches:

  • A batch is created only when a minimum number of items is available (default - 2 items).
  • A batch can be extended during a defined time period (default - 5 minutes - extendTime in the Assemble Batch step). It means that the batch size is increased, thus the sample size and rejection limit can be increased and more items will be sent for inspection.
  • A batch is marked as expired, if NOT all items from the sample are inspected during a defined time period (default - Task Expiration Time - expirationTime in the Assemble Batch step).

Accepting/rejecting batches

The main purpose and result of AutoQC is to provide an answer: accept or reject some batch. This decision is made after verifying all items in the sample (or a part of a sample in some cases described below).

A batch is rejected if the batch expired – not all sample items were inspected during expirationTime, OR if the rejection limit was reached when inspecting the sample.

A batch is accepted if the (number of correct sample items) ≥ (sample size - rejection limit).

If a batch is rejected, all remaining batch items (records in BP) go to the manual step.

Switching between inspection types

On the Assemble Batch step, a Batch Executor is created for each unique combination of automation Use Case and automated Human Campaign. One Batch Executor can contain multiple batches, which are distinguished by their unique ID.

Depending on the number of consecutively accepted or rejected batches, the batch executor can be switched to another inspection type:

note

In the Discontinue Inspection mode, ALL the upcoming items will be routed to human inspection without sampling.

It means that the Accuracy drops under the AutoQC acceptable level, and BP author will get an email notification.

The following diagram displays an example of a Batch Executor with 3 batches:

  • The Batch Executor has "production mode: 1". It means that the AutoQC process will be triggered only for production runs of the automated Human Campaign.
  • Note that batches have different sample size and rejection limit because of their batch size and inspection type.
  • Batch 1 and Batch 2 were rejected because they've reached the rejection limit, therefore Batch 3 has the TIGHTENED inspection type.
  • Batch 1 and Batch 2 were taken into consideration when switching to another inspection type (2 of 5 consecutive batches rejected - see the previous figure), therefore they have "finalized: 1"
note

Expired batches are not taken into consideration when switching to another inspection type.

AutoQC sub-process in the automation process

Cloud workers statistically verify the automated and delivered results match the predicted accuracy from this historical performance. AutoQC will automatically increase and decrease the amount of work sent to cloud workers to maintain the process performance at the requested acceptable level. AutoQC adapts during periods of repeated over-performance of the algorithms by decreasing human cloud work. Repeated periods of algorithm underperformance signal AutoML to retrain the algorithm.

  1. AutoQC can be enabled/disabled on the 1st BP step.
  2. After records pass Extract Information step via Success outcome, they are sent to the SQС sub-process.
  3. The first Is AutoQC enabled rule checks whether the quality control is needed. If the AutoQC was disabled on the Automation Settings step, all records are routed directly to the end without any inspection.
  4. On the Assemble Batch step, a Batch Executor is created for each unique combination of automation use case and automated human campaign. See the assembleBatch operation.
  5. Once the batch is assembled, number of records that corresponds to sample size are sent to Human task, the rest of the batch is sent to Wait for Inspection Results step. This condition is verified in the "Is inspection required?" by using AutoQC REST isInspectionRequired.
  6. After Workers have completed the human task, records are sent to Check Inspection Results step where machine answers are compared with human ones.
  7. The results of comparison are sent to the AutoQC application via REST addInspectionResult, and the AutoQC app provides inspection type and the batch status getBatchStatus:
    1. IN_PROGRESS
    2. ADDITIONAL_SAMPLE_REQUIRED
    3. ACCEPTED
    4. REJECTED
    5. EXPIRED
  8. Depending on the batch status, the rest of the batch records are routed to the End AutoQC step (if batch is accepted) or to the Manual Task. In case when additional sample is required, only one record is sent to the Manual Task.
  9. The last "Is inspection success?" rule routes records to an appropriate outcome depending on their inspection result ("_sys_sqc_inspection_result" column):
    1. Yes
    2. No
    3. There was no inspection

AutoQC application

AutoQC is an application that provides statistical sampling through REST API. It is used in WorkFusion Automation processes to verify automatically extracted information by Workers.

Tomcat configuration

AutoQC normally is deployed to separate server and its URL should be set in WF server Tomcat configuration:

<Resource name="jdbc/sqc" auth="Container" type="javax.sql.DataSource"
    maxActive="75" maxIdle="-1" maxWait="-1"
    username="root" password="root" driverClassName="com.mysql.jdbc.Driver"
    url="jdbc:mysql://localhost:3306/sqc?useLegacyDatetimeCode=false&amp;serverTimezone=UTC&amp;rewriteBatchedStatements=true"/>
 
<Environment name="sqc/applicationUrl" value="http://<host>:<port>/sqc-rest/api" type="java.lang.String"/>

In case AutoQC service is on another host then only "sqc/applicationUrl" JNDI required on WF Instance.

AutoQC REST API

Common parameters description:

  • sourceId is a parameter for multi-tenancy, use local in all cases;
  • identifier is a batch group identifier (statistic for this executor is calculated based on related to this identifier batches);
  • executorType helps to logically group executors. Currently used:
    • ML – machine learning;
    • AD – adjudication rule;
    • QC - quality check.

assembleBatch

  • URL: http[s]://<host>:<port>/sqc-rest/api/assembleBatch
  • Method: POST
  • Headers: Content-Type = application/json
  • Parameters:
{
         "sourceId":"sourceId",  
         "identifier":"identifier",
         "executorType":"ML",
         "name":"name",
         "numOfRecords": 100,
         "extendTime": 100000,
         "expirationTime": 10000000,
         "targetQuality":85
}
  • Response:
{
    "status": "OK",
    "errorMessage": "",
    "result": {
        "batchUuid": "9a64afb5-25cf-4d31-a23c-de2999e1a3d6",
        "inspectionType": "NORMAL"
    }
}

isInspectionRequired

  • URL: http[s]://<host>:<port>/sqc-rest/api/isInspectionRequired
  • Method: POST
  • Headers: –
  • Parameters: batchUuid = 9a64afb5-25cf-4d31-a23c-de2999e1a3d6
  • Response:
{
    "status": "OK",
    "errorMessage": "",
    "result": true
}

addInspectionResult

  • URL: http[s]://<host>:<port>/sqc-rest/api/addInspectionResult
  • Method: POST
  • Headers: –-
  • Parameters: batchUuid = 9a64afb5-25cf-4d31-a23c-de2999e1a3d6 result = 1
  • Response:
{
    "status": "OK",
    "errorMessage": "",
    "result": {
        "inspectionType": "NORMAL",
        "prevInspectionType": "NORMAL"
    }
}

getBatchStatus

  • URL: http[s]://<host>:<port>/sqc-rest/api/getBatchStatus
  • Method: POST
  • Headers: –-
  • Parameters: batchUuid = 9a64afb5-25cf-4d31-a23c-de2999e1a3d6
  • Response:
{
    "status": "OK",
    "errorMessage": "",
    "result": "IN_PROGRESS"
}

projectInfo

  • URL: http[s]://<host>:<port>/sqc-rest/api/projectInfo
  • Method: GET
  • Headers: –-
  • Parameters: –-
  • Response:
{
    "version":"2.0-SNAPSHOT",
    "repository":"https://svn.crowdcomputingsystems.com/svn/sqc",
    "path":"trunk",
    "revision":"143",
    "committedDate":"2014-12-23 06:56:50 +0000 (Tue, 23 Dec 2014)"
}

resetDiscontinue

This method returns batch executor from DISCONTINUE inspectionType to TIGHTENED and finalizes it's current batches.

  • URL: http[s]://<host>:<port>/sqc-rest/api/resetDiscontinue
  • Method: POST
  • Headers: –-
  • Parameters:
    • sourceId - BatchExecutor sourceId (String) (for example, "local");
    • identifier - BatchExecutor identifier (String) (for example, "84:a513ad77-65eb-4e7c-b472-128fd15794da");
    • productionMode - BatchExecutor productionMode (boolean) (e.g "1");-
  • Response:
{
    "version":"2.0-SNAPSHOT",
    "repository":"https://svn.crowdcomputingsystems.com/svn/sqc",
    "path":"trunk",
    "revision":"143",
    "committedDate":"2014-12-23 06:56:50 +0000 (Tue, 23 Dec 2014)"
}

Continuous sampling (not used in automation)

This type of AutoQC does not use batches, the statistics is calculated for all records, and the AutoQC application returns a response whether this particular record needs to be checked or not.

isNeedInspect

  • URL: http[s]://<host>:<port>/sqc-rest/api/isNeedInspect
  • Method: POST
  • Headers: Content-Type = application/json
  • Parameters:
{
    "sourceId":"sourceId",
    "identifier":"identifier",
    "executorType": "ML",
    "name": "name",
    "targetQuality":90
}

targetQuality should be in the range 0–100.

  • Response: true/false

addResult

  • URL: http[s]://<host>:<port>/sqc-rest/api/addResult
  • Method: POST
  • Headers: Content-Type = application/json
  • Parameters:
{
    "sourceId":"sourceId",
    "identifier":"identifier",
    "result": 0,
    "targetQuality":90
}

targetQuality should be in the range 0–100. result should be 0 or 1.

Average results for experiments

Each experiment was performed 10 times.

Expand to learn results

Experiment12345
PRECONDITIONS
Total batches processed100100100100100
Min batch size11111
Max batch size10001000100010001000
Avg batch size446547407368402
Min batch quality0,700,750,800,850,90
Max batch quality0,950,950,950,950,95
Target quality0,900,900,900,900,90
RESULTS
Accepted batches4,77,939,482,199,7
Rejected batches95,392,160,617,90,3
Total records4468054740407003689040220
Correct records44530,554422,239886,534418,537402,4
Incorrect records149,5317,8813,52471,52817,6
Manually processed43567,751892,134142,812186,02788,8
Manually processed, %0,980,950,840,330,07
Actual results quality0,99430,99170,95630,92720,9313

← Prepare datasetModels overview →