10.2

Edit

Adjudication Rules

important

Starting from IA Cloud Enterprise v10.2, you can only use the (1+0) Adjudication Rule. Other Adjudication Rules are obsolete and are to be removed from the Control Tower user interface in further releases.

Adjudication is a process that ensures the quality of answers provided by human workers. Adjudication rules help one configure the process.

On each platform instance, there are several adjudication rules created by default. You can access them from the Rules page. In addition, you can create your own rules.

Every Manual Task you create has an Adjudication Rule associated with it. To view the rule or change it if needed, go to Run Task > Advanced Options > Adjudication.

Adjudication Rule parameters

Majority value

Determines how many workers need to agree in their answers for the assignment to be accepted and the task to be closed.

Maximum assignments

Determines the maximum number of assignments to be created during Adjudication. If a worker does not provide the required majority and the task needs to be extended by posting more assignments, the Intelligent Automation Cloud will not exceed the value specified in this parameter when posting more assignments.

Evaluation frequency

The rule is also known as the CHECK_EVERY rule.

Normal accuracy % vs Super accuracy %

Currently, the Intelligent Automation Cloud has two rankings it assigns to workers: Normal and Super. Usually, Normal workers are those who are known to perform adequately well in a given task. Super workers are those who are known to perform exceptionally well in a given task. These two parameters allow you to determine what constitutes Normal and Super.

For example, if you set Normal Accuracy to 75%, a worker gets the Normal ranking provided he gives at least 75% correct answers to gold tasks. And if you set the Super Accuracy to 90%, the system only grants workers Super Accuracy if they respond correctly to at least 90% of the gold tasks presented to them. The parameter is sometimes referred to as UPPER_LIMIT and LOWER_LIMIT.

Normal Accuracy score vs Super Accuracy score

On some endpoints, like Mechanical Turk, a task needs to have a score associated with it. Based on the Gold Accuracy score, a worker is assigned a ranking. Then based on the ranking, the worker is assigned a score accounting for Normal and Super Accuracy scores. Essentially, the two parameters define the score granted to workers based on the ranking they achieved.

Adjudication Rules can contain other rules that verify worker statistics and execute actions based on the results. See the rules below.

Retract worker answers

The functionality is intended to exclude answers from the final result (for example, if a worker is a cheater). If a task contains retracted answers, the majority and confidence are recalculated. The Retract button is available only for Manual Tasks that are not Qualification Tasks.

You can retract answers in the following manner:

  • Manually via Task > View Results > Workers tab > Retract Worker Answers. A worker can continue to work on tasks in the same run. The answers given by the worker after the manual retract action won't be retracted automatically.
  • By the Adjudication Rule. If answers were retracted by the rule, the future answers of the worker in this run are also retracted and rejected. The retracted assignment is not included in the Max assignment limit count. If a task contains more than one assignment, an additional assignment is created instead of the retracted one.

Retract rule example

rule "Retract less 7 sec"
agenda-group "calculation"
dialect "mvel"
salience 20
no-loop
    when
        $ctx:RuleContext();
        $rac:RuleAssigmentContext();
        eval($rac.assignment.submitTime.time - $rac.assignment.acceptTime.time <= 7000);
    then
       $ctx.retractWorker($rac.assignment.WorkerNativeId);
       $ctx.logExecutedRule(kcontext.getRule().getName());
end

rule "Retract already retracted"
agenda-group "calculation"
dialect "mvel"
salience 20
no-loop
    when
        $ctx:RuleContext();
        $rac:RuleAssigmentContext(WorkerRetracted == true);       
    then
       $ctx.retractWorker($rac.assignment.WorkerNativeId);
       $ctx.logExecutedRule(kcontext.getRule().getName());
end
rule "Retract low confidence Worker"
agenda-group "calculation"
dialect "mvel"
salience 20
no-loop
    when
        $ctx:RuleContext();
        $confidencesForAnswer: java.util.Map() from $ctx.WorkerConfidences.values();
        $entry: Map.Entry(value < 0.7) from $confidencesForAnswer.entrySet();
    then
       $ctx.retractWorker($entry.key);
       $ctx.logExecutedRule(kcontext.getRule().getName());
end

rule "Extend HIT when all assignments retracted"
agenda-group "calculation"
dialect "mvel"
salience 50
no-loop
    when
        $ctx:RuleContext(initialized == true);
        eval($ctx.totalAssignmentCount < $ctx.properties.MAX_ASSIGNMENT_LIMIT);
        eval($ctx.assignments.empty);
    then
        $ctx.setExtendHit(true);
        $ctx.logExecutedRule(kcontext.getRule().getName());
end

Disqualify workers

The rule allows to disqualify workers if their Performance, Accuracy, or Gold Accuracy do not meet the specified requirements.

Retract worker when response time is too high

The part marks assignments as Retracted in the database if a worker answers too fast. For the assignment to be marked this way, the following criteria should be met:

  • More than one answer from other workers. The number of answers can be specified in the Time_Check_threshold parameter.
  • The time spent on the task should be lower than the time median of other workers divided by two. All answers greater than the 90th percentile are excluded from the median counting.
($aggregatedWorkerStatistics.otherWorkersAnswerCount \> 1 &&

$rac.responseTime \< $aggregatedWorkerStatistics.otherWorkersTaskResponseTimeMedian / 2)

Retract round when Gold Accuracy is low

The part retracts a round for a worker. If the worker has answers marked as retracted in the database, such answers are retracted in the user interface, and HIT is extended according to the rule.

In this case, for the round to be retracted, the following criteria should be met:

  • The task should have more than 0 completed HITs.
  • The Retract_Round_Size parameter should be reached.
  • Total Golds in the run should be greater than 0.
  • The Worker GA*100 should be lower than 90, where 90 is the parameter you can specify in Gold_Accuracy_Level_Limit for the retract action.
($ctx.completeHitCount > 0 && ($ctx.completeHitCount  %  2) == 0 &&
($gold.size() > 0 || $runStatistic.totalGold > 0 ) && RuleContextUtils.getTaskWorkerGoldAccuracy($rac) * 100 < 90)

Retract round when Accuracy is low

The part retracts a round when the worker accuracy is low.

For the round to be retracted, the following criteria should be met:

  • The task should have more than 0 completed HITs.
  • The Retract_Round_Size parameter should be reached.
  • The total majority count should be greater than 0.
  • The Worker Accuracy\100 should be lower than 90, where 90 is the parameter you can specify in the Accuracy_Level_Limit for the retract action.
($ctx.completeHitCount > 0 && ($ctx.completeHitCount  %  2) == 0 &&
$runStatistic.totalMajorityCount > 0 && RuleContextUtils.getTaskWorkerAccuracy($rac) * 100 < 90)

Disqualify worker when Gold Accuracy is low

This part of the rule disqualifies workers and lowers their qualification. Therefore, the workers cannot continue to give answers to tasks.

For a worker to be disqualified, the following criteria should be met:

  • The worker should have more than one retracted round.
  • Other workers should have more than 0 gold answers.
  • The worker's Gold Accuracy should be lower than the other workers' Gold Accuracy median*0.9.

The Qualification Score to be assigned to the worker after the disqualification can be specified in the Disqualification_score parameter.

(($rs.roundRetracted && $rac.retractedRounds + 1 > 1) || (!$rs.roundRetracted && $rac.retractedRounds > 1) )
and
eval($aggregatedWorkerStatistics.otherWorkersGoldAnswerCount > 0 &&
RuleContextUtils.getTaskWorkerGoldAccuracy($rac) < $aggregatedWorkerStatistics.otherWorkersGoldAccuracyMedian * 0.9)

Disqualify worker when response time is too high

For a worker to be disqualified, the following criteria should be met:

  • The worker should have more than one retracted round.
  • The worker task response median should be lower than the other workers' task response median/2.

The Qualification Score to be assigned to the worker after the disqualification can be specified in the Disqualification_score parameter.

(($rs.roundRetracted && $rac.retractedRounds + 1 > 1) || (!$rs.roundRetracted && $rac.retractedRounds > 1) )      and  eval($aggregatedWorkerStatistics.WorkerTaskResponseTimeMedian < $aggregatedWorkerStatistics.otherWorkersTaskResponseTimeMedian / 2)

Disqualify worker when performance is low

For a worker to be disqualified, the following criteria should be met:

  • The worker's total task number should be greater than one.
  • The number of retracted tasks for the worker in the database should be greater than total*50/100.
($aggregatedWorkerStatistics.totalTaskCount > 1)
and
(eval($rs.assignmentRetracted && $rac.retractedTasks + 1 > ($aggregatedWorkerStatistics.totalTaskCount * 50) / 100) or
eval(!$rs.assignmentRetracted && $rac.retractedTasks > ($aggregatedWorkerStatistics.totalTaskCount * 50) / 100) )
← PreviousNext →