Adjudication Rules

Adjudication is a process that ensures the quality of answers provided by human workers. Adjudication rules help one configure the process.

On each platform instance, there are several adjudication rules created by default. You can access them from the Rules page. In addition, you can create your own rules.

Every Manual Task you create has an Adjudication Rule associated with it. To view the rule or change it if needed, go to Run Task > Advanced Options > Adjudication.

The Adjudication Rule configuration in the screenshot above tells the Intelligent Automation Cloud to use a minimum of two Assignments, a maximum of three Assignments, and the 2 + 1, pay all rule for the particular task.

This means that the platform initially posts two assignments of the task on the endpoint (for example, WorkSpace or Amazon Mechanical Turk). If there are multiple Records in the input file, it posts two assignments for each of these records. Then, as workers complete these assignments, the platform checks if the answers match between the two assignments. If yes, the task is accepted, and no further assignments are posted. Otherwise, a third assignment is posted. When the third assignment is submitted, the platform compares its result with that of the previous two. If it finds a match, it accepts the answers that were matching. If there is still no match, no further assignments are posted since the Max # of Assignments field was set to 3. The output file (snapshot) contains the answers provided by the three Workers and an indication that no confidence was found.

You can configure Adjudication Rules using Rule Parameters in the Basic View and the actual Rules Code in the Advanced View.

Adjudication rule example

package com.freedomoss.requester;
#list any import classes here.
import com.freedomoss.objective.model.RuleContext;
import com.freedomoss.objective.model.RuleContext.MajorityType;
import com.freedomoss.objective.model.RuleAssigmentContext;
import com.freedomoss.objective.model.RuleQuestionContext;
import com.freedomoss.requester.model.AwsHitQuestion;
import com.freedomoss.requester.model.AwsHitQuestionItem;
import com.freedomoss.requester.model.AwsHitQuestionAnswerItem;
import org.slf4j.Logger;
import java.util.ArrayList;
import java.util.List;
import java.util.Set;
import java.util.Map;
import java.util.Iterator;
#declare any global variables here
global RuleContext source
global Logger log
global Map params

rule "Rule context initialization"
auto-focus true
no-loop
dialect "mvel"
agenda-group "initialization-group"
    when
        $ctx:RuleContext(initialized == false);
    then
        # set parameters
        $ctx.properties[RuleContext.MAJORITY_TYPE] =  RuleContext.MajorityType.COUNT;
        $ctx.properties[RuleContext.MAJORITY_VALUE] = new Integer(2);
        $ctx.properties[RuleContext.MAX_ASSIGNMENT_LIMIT] = new Integer(3);
        $ctx.properties[RuleContext.MAJORITY_HIT_THRESHOLD] = new Double(100/100);
        $ctx.properties[RuleContext.ASSIGNMENT_APPROVE_THRESHOLD] = new Double(0.5);
        # insert processed facts into memory
        $ctx.updateWorkingMemory();
        # move to business rules
        kcontext.getKnowledgeRuntime().getAgenda().getAgendaGroup("calculation").setFocus();
        $ctx.logExecutedRule(kcontext.getRule().getName());
end
rule "Worst accuracy rule - Evaluate every 5 gold question and set Accuracy based qualification score to 70 if gold accuracy goes up < 75 percents"
agenda-group "calculation"
dialect "mvel"
salience 250
no-loop
 when
     $ctx:RuleContext(initialized == true)
     $rac:RuleAssigmentContext($campaignStatistic:campaignStatistic, runStatistic.totalGoldQuestions > 0)
     #evaluate run average response time
     eval($campaignStatistic.totalGoldQuestions  >  0 &&
         ($campaignStatistic.totalGoldQuestions  %  5) == 0 && $campaignStatistic.goldAccuracy < (75/100))
 then
     $rac.grandQualification(RuleContext.ACCURACY_BASED_QUALIFICATION, 70);
     $ctx.logExecutedRule(kcontext.getRule().getName());
end
rule "Normal accuracy rule - Evaluate every 5 gold question and set Accuracy based qualification score to 80 if gold accuracy goes up >= 75 percents and < 90"
agenda-group "calculation"
dialect "mvel"
salience 250
no-loop
 when
     $ctx:RuleContext(initialized == true)
     $rac:RuleAssigmentContext($campaignStatistic:campaignStatistic, runStatistic.totalGoldQuestions > 0)
     #evaluate run average response time
     eval($campaignStatistic.totalGoldQuestions  >  0 &&
         ($campaignStatistic.totalGoldQuestions  %  5) == 0 &&
          $campaignStatistic.goldAccuracy >= (75/ 100) && $campaignStatistic.goldAccuracy < (90 / 100))
 then
     $rac.grandQualification(RuleContext.ACCURACY_BASED_QUALIFICATION, 80);
     $ctx.logExecutedRule(kcontext.getRule().getName());
end
rule "Super accuracy rule - Evaluate every 5 gold question and set Accuracy based qualification score to 95 if gold accuracy goes up >= 90 percents"
agenda-group "calculation"
dialect "mvel"
salience 250
no-loop
 when
     $ctx:RuleContext(initialized == true)
     $rac:RuleAssigmentContext($campaignStatistic:campaignStatistic, runStatistic.totalGoldQuestions > 0)
     #evaluate run average response time
     eval($campaignStatistic.totalGoldQuestions  >  0 &&
         ($campaignStatistic.totalGoldQuestions  %  5) == 0 &&
          $campaignStatistic.goldAccuracy >= (90 / 100) )
 then
     $rac.grandQualification(RuleContext.ACCURACY_BASED_QUALIFICATION, 95);
     $ctx.logExecutedRule(kcontext.getRule().getName());
end
rule "0. Check, if exist majority"
agenda-group "calculation"
dialect "mvel"
salience 100
no-loop
    when
        $ctx:RuleContext (initialized == true, $gold:gold, $threshold:properties.MAJORITY_HIT_THRESHOLD)
        eval($ctx.majorityWithoutGold().size() >= ($ctx.questions.size() - $gold.size()) *  $threshold)
    then
        insert(new String("MAJORITY_FOUND"));
        $ctx.logExecutedRule(kcontext.getRule().getName());
end
rule "1. Approve question by majority"
agenda-group "calculation"
dialect "mvel"
salience 90
no-loop
    when
        ((String(toString == "MAJORITY_FOUND") and  $ctx:RuleContext (initialized == true))
        or
        (not String(toString == "MAJORITY_FOUND") and  $ctx:RuleContext (initialized == true, assignments.size == properties.MAX_ASSIGNMENT_LIMIT)))
        $rqc:RuleQuestionContext()
    then
        $rqc.approveQuestion($ctx.majority());
        $ctx.logExecutedRule(kcontext.getRule().getName(), $rqc);
end
rule "3. Approve assignment (always)"
agenda-group "calculation"
dialect "mvel"
salience 50
no-loop
    when
        $ctx:RuleContext(initialized == true)
        $rac:RuleAssigmentContext()
    then
        insert(new String("ASSIGNMENT_PROCESSED"));
        $ctx.addApproved($rac);
        $ctx.logExecutedRule(kcontext.getRule().getName(), $rac);
end

rule "5. Extend HIT"
agenda-group "calculation"
dialect "mvel"
salience 50
no-loop
    when
        not String(toString == "MAJORITY_FOUND")
        $ctx:RuleContext(initialized == true, assignments.size < properties.MAX_ASSIGNMENT_LIMIT);
    then
        $ctx.setExtendHit(true);
        $ctx.logExecutedRule(kcontext.getRule().getName());
end
rule "6. Dispose HIT, no majority"
agenda-group "calculation"
dialect "mvel"
salience 30
no-loop
    when
        not String(toString == "MAJORITY_FOUND");
        $ctx:RuleContext(initialized == true, assignments.size >= properties.MAX_ASSIGNMENT_LIMIT);
        $rac:RuleAssigmentContext();
    then
        $ctx.addApproved($rac);
        $ctx.setDisposeHit(true);
        $ctx.logExecutedRule(kcontext.getRule().getName());
end

Adjudication Rule parameters

Majority value

Determines how many workers need to agree in their answers for the assignment to be accepted and the task to be closed. For example, if the parameter is set to 2, the Intelligent Automation Cloud creates two assignments (task instances) on the endpoint (WorkSpace, Mechanical Turk, and so on). If workers submit the same answers for both assignments, the task is closed, meaning it is not posted again, and the workers are paid. But if it returns different answers, one more assignment is created. The platform continues posting assignments until two workers provide the same answer.

Increasing this value ensures the quality of your work subsequently, but it may also result in more assignments being created and thus more Workers needing to be paid.

Maximum assignments

Determines the maximum number of assignments to be created during Adjudication. If a worker does not provide the required majority and the task needs to be extended by posting more assignments, the Intelligent Automation Cloud will not exceed the value specified in this parameter when posting more assignments.

For example, if the Majority value is 2, and more assignments need to be posted to reach that majority, whereas Maximum assignments is set to 4, then at most four assignments of the task are posted. If, after the four assignments, the majority still has not been reached, the task is closed.

Increasing this value ensures the quality of your work subsequently, but it may also result in more assignments being created and thus more Workers needing to be paid.

Evaluation frequency

The rule is also known as the CHECK_EVERY rule. In terms of Ongoing Qualifications, it determines after how many Gold Tasks a worker is evaluated on the task.

For example, if the parameter is set to 5, the system evaluates the worker's performance and updates its score after every five Gold Tasks.

Normal accuracy % vs Super accuracy %

Currently, the Intelligent Automation Cloud has two rankings it assigns to workers: Normal and Super. Usually, Normal workers are those who are known to perform adequately well in a given task. Super workers are those who are known to perform exceptionally well in a given task. These two parameters allow you to determine what constitutes Normal and Super.

For example, if you set Normal Accuracy to 75%, a worker gets the Normal ranking provided he gives at least 75% correct answers to gold tasks. And if you set the Super Accuracy to 90%, the system only grants workers Super Accuracy if they respond correctly to at least 90% of the gold tasks presented to them. The parameter is sometimes referred to as UPPER_LIMIT and LOWER_LIMIT.

Normal Accuracy score vs Super Accuracy score

On some endpoints, like Mechanical Turk, a task needs to have a score associated with it. This score is different from the Gold Accuracy one, which is the percentage of the correct worker answers on Gold Data tasks.

Based on the Gold Accuracy score, the worker gets a ranking as described above. Then based on the ranking, the worker is assigned a score accounting for Normal and Super Accuracy scores. Essentially, the two parameters define the score granted to workers based on the ranking they achieved. So, for example, if they achieve a ranking of Super and the Super Accuracy parameter is set to 90, they are assigned a score of 90 on the Qualification associated with the task.

Adjudication Rules can contain other rules that verify worker statistics and execute actions based on the results. See the rules below.

Retract Worker Answers

The functionality is intended to exclude answers from the final result (for example, if a worker is a cheater). If a task contains retracted answers, the majority and confidence are recalculated. The Retract button is available only for Manual Tasks that are not Qualification Tasks.

You can retract answers in the following manner:

Manually via Task > View Results > Workers tab > Retract Worker Answers. A worker can continue to work on tasks in the same run. The answers given by the worker after the manual retract action won't be retracted automatically.
By the Adjudication Rule. If answers were retracted by the rule, the future answers of the worker in this run are also retracted and rejected. The retracted assignment is not included in the Max assignment limit count. If a task contains more than one assignment, an additional assignment is created instead of the retracted one.

Retract rule example

rule "Retract less 7 sec"
agenda-group "calculation"
dialect "mvel"
salience 20
no-loop
    when
        $ctx:RuleContext();
        $rac:RuleAssigmentContext();
        eval($rac.assignment.submitTime.time - $rac.assignment.acceptTime.time <= 7000);
    then
       $ctx.retractWorker($rac.assignment.WorkerNativeId);
       $ctx.logExecutedRule(kcontext.getRule().getName());
end

rule "Retract already retracted"
agenda-group "calculation"
dialect "mvel"
salience 20
no-loop
    when
        $ctx:RuleContext();
        $rac:RuleAssigmentContext(WorkerRetracted == true);       
    then
       $ctx.retractWorker($rac.assignment.WorkerNativeId);
       $ctx.logExecutedRule(kcontext.getRule().getName());
end
rule "Retract low confidence Worker"
agenda-group "calculation"
dialect "mvel"
salience 20
no-loop
    when
        $ctx:RuleContext();
        $confidencesForAnswer: java.util.Map() from $ctx.WorkerConfidences.values();
        $entry: Map.Entry(value < 0.7) from $confidencesForAnswer.entrySet();
    then
       $ctx.retractWorker($entry.key);
       $ctx.logExecutedRule(kcontext.getRule().getName());
end

rule "Extend HIT when all assignments retracted"
agenda-group "calculation"
dialect "mvel"
salience 50
no-loop
    when
        $ctx:RuleContext(initialized == true);
        eval($ctx.totalAssignmentCount < $ctx.properties.MAX_ASSIGNMENT_LIMIT);
        eval($ctx.assignments.empty);
    then
        $ctx.setExtendHit(true);
        $ctx.logExecutedRule(kcontext.getRule().getName());
end

Disqualify workers

The rule allows to disqualify workers if their Performance, Accuracy, or Gold Accuracy do not meet the specified requirements.

Disqualify rule example

package com.freedomoss.requester;

#list any import classes here.
import com.freedomoss.objective.model.RuleContext;
import com.freedomoss.objective.model.RuleContext.MajorityType;
import com.freedomoss.objective.model.RuleAssigmentContext;
import com.freedomoss.objective.model.RuleQuestionContext;
import com.freedomoss.requester.model.AwsHitQuestion;
import com.freedomoss.requester.model.AwsHitQuestionItem;
import com.freedomoss.requester.model.AwsHitQuestionAnswerItem;
import com.freedomoss.objective.model.RuleContextUtils;
import com.freedomoss.objective.facts.RetractState;

import org.slf4j.Logger;
import java.util.ArrayList;
import java.util.List;
import java.util.Set;
import java.util.Map;
import java.util.Iterator;

#declare any global variables here
global RuleContext source
global Logger log
global Map params


rule "Rule context initialization"
auto-focus true
no-loop
dialect "mvel"
agenda-group "initialization-group"
  when
        $ctx:RuleContext(initialized == false);
  then
        # set parameters

        $ctx.properties[RuleContext.MAJORITY_TYPE] =  RuleContext.MajorityType.COUNT;
        $ctx.properties[RuleContext.MAJORITY_VALUE] = new Integer(2);

        $ctx.properties[RuleContext.MAX_ASSIGNMENT_LIMIT] = new Integer(3);
        $ctx.properties[RuleContext.MAJORITY_HIT_THRESHOLD] = new Double(100/100);
        $ctx.properties[RuleContext.ASSIGNMENT_APPROVE_THRESHOLD] = new Double(0.5);

        # insert processed facts into memory
        $ctx.updateWorkingMemory();
        # move to business rules
        kcontext.getKnowledgeRuntime().getAgenda().getAgendaGroup("calculation").setFocus();
        $ctx.logExecutedRule(kcontext.getRule().getName());
end

rule "Worst accuracy rule - Evaluate every 5 gold question and set Accuracy based qualification score to 70 if gold accuracy goes up < 75 percents"
agenda-group "calculation"
dialect "mvel"
salience 250
no-loop
     when
         $ctx:RuleContext(initialized == true)
         $rac:RuleAssigmentContext($campaignStatistic:campaignStatistic)
         #evaluate run average response time
         eval($campaignStatistic.totalGoldQuestions  >  0 &&
             ($campaignStatistic.totalGoldQuestions  %  5) == 0 && $campaignStatistic.goldAccuracy < (75/100))
     then
         $rac.grantQualification(RuleContext.ACCURACY_BASED_QUALIFICATION, 70);
         $ctx.logExecutedRule(kcontext.getRule().getName());
end

rule "Normal accuracy rule - Evaluate every 5 gold question and set Accuracy based qualification score to 80 if gold accuracy goes up >= 75 percents and < 90"
agenda-group "calculation"
dialect "mvel"
salience 250
no-loop
     when
         $ctx:RuleContext(initialized == true)
         $rac:RuleAssigmentContext($campaignStatistic:campaignStatistic)
         #evaluate run average response time
         eval($campaignStatistic.totalGoldQuestions  >  0 &&
             ($campaignStatistic.totalGoldQuestions  %  5) == 0 &&
              $campaignStatistic.goldAccuracy >= (75/ 100) && $campaignStatistic.goldAccuracy < (90 / 100))
     then
         $rac.grantQualification(RuleContext.ACCURACY_BASED_QUALIFICATION, 80);
         $ctx.logExecutedRule(kcontext.getRule().getName());
end

rule "Super accuracy rule - Evaluate every 5 gold question and set Accuracy based qualification score to 95 if gold accuracy goes up >= 90 percents"
agenda-group "calculation"
dialect "mvel"
salience 250
no-loop
     when
         $ctx:RuleContext(initialized == true)
         $rac:RuleAssigmentContext($campaignStatistic:campaignStatistic)
         #evaluate run average response time
         eval($campaignStatistic.totalGoldQuestions  >  0 &&
             ($campaignStatistic.totalGoldQuestions  %  5) == 0 &&
              $campaignStatistic.goldAccuracy >= (90 / 100) )
     then
         $rac.grantQualification(RuleContext.ACCURACY_BASED_QUALIFICATION, 95);
         $ctx.logExecutedRule(kcontext.getRule().getName());
end

rule "0. Check, if exist majority"
agenda-group "calculation"
dialect "mvel"
salience 100
no-loop
     when
          $ctx:RuleContext (initialized == true, $gold:gold, $threshold:properties.MAJORITY_HIT_THRESHOLD)
          eval($ctx.assignments.size() > 0 && $ctx.majorityWithoutGold().size() >= ($ctx.questions.size() - $gold.size()) *  $threshold)
     then
          insert(new String("MAJORITY_FOUND"));
          $ctx.logExecutedRule(kcontext.getRule().getName());
end

rule "1. Approve question by majority"
agenda-group "calculation"
dialect "mvel"
salience 90
no-loop
    when
        ((String(toString == "MAJORITY_FOUND") and  $ctx:RuleContext (initialized == true))
       or
        (not String(toString == "MAJORITY_FOUND") and  $ctx:RuleContext (initialized == true, assignments.size == properties.MAX_ASSIGNMENT_LIMIT)))
        $rqc:RuleQuestionContext()
    then
        $rqc.approveQuestion($ctx.majority());
        $ctx.logExecutedRule(kcontext.getRule().getName(), $rqc);
end

rule "3. Approve assignment (always)"
agenda-group "calculation"
dialect "mvel"
salience 50
no-loop
     when
          $ctx:RuleContext(initialized == true)
          $rac:RuleAssigmentContext()
     then
         insert(new String("ASSIGNMENT_PROCESSED"));
         $ctx.addApproved($rac);
         $ctx.logExecutedRule(kcontext.getRule().getName(), $rac);
end


rule "5. Extend HIT"
agenda-group "calculation"
dialect "mvel"
salience 50
no-loop
      when
          not String(toString == "MAJORITY_FOUND")
          $ctx:RuleContext(initialized == true, assignments.size < properties.MAX_ASSIGNMENT_LIMIT);
      then
          $ctx.setExtendHit(true);
          $ctx.logExecutedRule(kcontext.getRule().getName());
end

rule "6. Dispose HIT, no majority"
agenda-group "calculation"
dialect "mvel"
salience 30
no-loop
      when
          not String(toString == "MAJORITY_FOUND");
          $ctx:RuleContext(initialized == true, assignments.size >= properties.MAX_ASSIGNMENT_LIMIT);
          $rac:RuleAssigmentContext();
      then
          $ctx.addApproved($rac);
          $ctx.setDisposeHit(true);
          $ctx.logExecutedRule(kcontext.getRule().getName());
end


rule "Init retract states"
agenda-group "calculation"
dialect "mvel"
no-loop
     when
         $ctx:RuleContext(initialized == true)
         $rac:RuleAssigmentContext()
     then
         insert(new RetractState($rac.assignmentId), true);
end


rule "7. Retract Worker when response time is too high"
agenda-group "calculation"
dialect "mvel"
salience 20
no-loop
      when
          $ctx:RuleContext(initialized == true)
          $rac:RuleAssigmentContext($aggregatedWorkerStatistics:aggregatedWorkerStatistics)
          $rs:RetractState(assignmentId == $rac.assignmentId, assignmentRetracted == false)

          eval($aggregatedWorkerStatistics.otherWorkersAnswerCount > 1 &&
               $rac.responseTime < $aggregatedWorkerStatistics.otherWorkersTaskResponseTimeMedian / 2)
      then
          log.info("Retract assignment for Worker with high response time: " + $rac.nativeWorkerId);
          $ctx.retractAssignment($rac.nativeWorkerId);
          modify($rs) {assignmentRetracted = true};
          $ctx.logExecutedRule(kcontext.getRule().getName());
end


rule "8. Retract round when gold accuracy is low"
agenda-group "calculation"
dialect "mvel"
salience 19
no-loop
     when
         $ctx:RuleContext(initialized == true, $gold:gold)
         $rac:RuleAssigmentContext($runStatistic:runStatistic)
         $rs:RetractState(assignmentId == $rac.assignmentId, roundRetracted == false)

         eval($ctx.completeHitCount > 0 && ($ctx.completeHitCount  %  2) == 0 &&
               ($gold.size() > 0 || $runStatistic.totalGold > 0 ) && RuleContextUtils.getTaskWorkerGoldAccuracy($rac) * 100 < 90)
     then
         log.info("Retract round for Worker with low gold accuracy: " + $rac.nativeWorkerId);
         $ctx.retractRound($rac.nativeWorkerId);
         modify($rs) {roundRetracted = true};
         $ctx.logExecutedRule(kcontext.getRule().getName());
end

rule "9. Retract round when accuracy is low"
agenda-group "calculation"
dialect "mvel"
salience 18
no-loop
     when
         $ctx:RuleContext(initialized == true)
         $rac:RuleAssigmentContext($runStatistic:runStatistic)
         $rs:RetractState(assignmentId == $rac.assignmentId, roundRetracted == false)

         eval($ctx.completeHitCount > 0 && ($ctx.completeHitCount  %  2) == 0 &&
               $runStatistic.totalMajorityCount > 0 && RuleContextUtils.getTaskWorkerAccuracy($rac) * 100 < 90)
     then
         log.info("Retract round for Worker with low accuracy: " + $rac.nativeWorkerId);
         $ctx.retractRound($rac.nativeWorkerId);
         modify($rs) {roundRetracted = true};
         $ctx.logExecutedRule(kcontext.getRule().getName());
end

rule "10. Disqualify when when gold accuracy is low"
agenda-group "calculation"
dialect "mvel"
salience 17
no-loop
     when
         $ctx:RuleContext(initialized == true, $gold:gold)
         $rac:RuleAssigmentContext($aggregatedWorkerStatistics:aggregatedWorkerStatistics)
         $rs:RetractState(assignmentId == $rac.assignmentId)

         eval(($rs.roundRetracted && $rac.retractedRounds + 1 > 1) || (!$rs.roundRetracted && $rac.retractedRounds > 1) )      
           and
         eval($aggregatedWorkerStatistics.otherWorkersGoldAnswerCount > 0 &&
              RuleContextUtils.getTaskWorkerGoldAccuracy($rac) < $aggregatedWorkerStatistics.otherWorkersGoldAccuracyMedian * 0.9)
     then
         disqualifyWorker($rac, 65, log);
         $ctx.logExecutedRule(kcontext.getRule().getName());
end

rule "11. Disqualify when response time is too high"
agenda-group "calculation"
dialect "mvel"
salience 16
no-loop
     when
         $ctx:RuleContext(initialized == true)
         $rac:RuleAssigmentContext($aggregatedWorkerStatistics:aggregatedWorkerStatistics)
         $rs:RetractState(assignmentId == $rac.assignmentId)

         eval(($rs.roundRetracted && $rac.retractedRounds + 1 > 1) || (!$rs.roundRetracted && $rac.retractedRounds > 1) )    
           and
         eval($aggregatedWorkerStatistics.WorkerTaskResponseTimeMedian < $aggregatedWorkerStatistics.otherWorkersTaskResponseTimeMedian / 2)
     then
         disqualifyWorker($rac, 65, log);
         $ctx.logExecutedRule(kcontext.getRule().getName());
end

rule "12. Disqualify when performance is low"
agenda-group "calculation"
dialect "mvel"
salience 15
no-loop
     when
         $ctx:RuleContext(initialized == true)
         $rac:RuleAssigmentContext($aggregatedWorkerStatistics:aggregatedWorkerStatistics)
         $rs:RetractState(assignmentId == $rac.assignmentId)

         eval($aggregatedWorkerStatistics.totalTaskCount > 1)
            and
         (eval($rs.assignmentRetracted && $rac.retractedTasks + 1 > ($aggregatedWorkerStatistics.totalTaskCount * 50) / 100) or
         eval(!$rs.assignmentRetracted && $rac.retractedTasks > ($aggregatedWorkerStatistics.totalTaskCount * 50) / 100) )
     then
         disqualifyWorker($rac, 65, log);
         $ctx.logExecutedRule(kcontext.getRule().getName());
end


function void disqualifyWorker(RuleAssigmentContext rac, int score, Logger log) {
    RuleContext rc = rac.getParent();

    log.info("Disqualifying Worker: " + rac.getNativeWorkerId());
    String autoGrantedQualificationUUID = RuleContextUtils.getAutoGrantedQualificationUuid(rc);

    if (autoGrantedQualificationUUID != null) {   
        log.info("Changing  auto granted qualification " + autoGrantedQualificationUUID + " score to " + score);
        rac.forceGrantQualification(autoGrantedQualificationUUID, score);
    } else {
        String accuracyQualificationUUID = RuleContextUtils.getAccuracyBasedQualificationUuid(rc);
        if(accuracyQualificationUUID != null) {
            log.info("Changing accuracy based qualification " + accuracyQualificationUUID + " score to " + score);
            rac.forceGrantQualification(accuracyQualificationUUID, score);
        }
    }
}

Retract worker when response time is too high

The part marks assignments as Retracted in the database if a worker answers too fast. For the assignment to be marked this way, the following criteria should be met:

More than one answer from other workers. The number of answers can be specified in the Time_Check_threshold parameter.
The time spent on the task should be lower than the time median of other workers divided by two. All answers greater than the 90th percentile are excluded from the median counting.

($aggregatedWorkerStatistics.otherWorkersAnswerCount \> 1 &&

$rac.responseTime \< $aggregatedWorkerStatistics.otherWorkersTaskResponseTimeMedian / 2)

Retract round when Gold Accuracy is low

The part retracts a round for a worker. If the worker has answers marked as retracted in the database, such answers are retracted in the user interface, and HIT is extended according to the rule.

In this case, for the round to be retracted, the following criteria should be met:

The task should have more than 0 completed HITs.
The Retract_Round_Size parameter should be reached.
Total Golds in the run should be greater than 0.
The Worker GA*100 should be lower than 90, where 90 is the parameter you can specify in Gold_Accuracy_Level_Limit for the retract action.

($ctx.completeHitCount > 0 && ($ctx.completeHitCount  %  2) == 0 &&
($gold.size() > 0 || $runStatistic.totalGold > 0 ) && RuleContextUtils.getTaskWorkerGoldAccuracy($rac) * 100 < 90)

Retract round when Accuracy is low

The part retracts a round when the worker accuracy is low.

For the round to be retracted, the following criteria should be met:

The task should have more than 0 completed HITs.
The Retract_Round_Size parameter should be reached.
The total majority count should be greater than 0.
The Worker Accuracy\100 should be lower than 90, where 90 is the parameter you can specify in the Accuracy_Level_Limit for the retract action.

($ctx.completeHitCount > 0 && ($ctx.completeHitCount  %  2) == 0 &&
$runStatistic.totalMajorityCount > 0 && RuleContextUtils.getTaskWorkerAccuracy($rac) * 100 < 90)

Disqualify worker when Gold Accuracy is low

This part of the rule disqualifies workers and lowers their qualification. Therefore, the workers cannot continue to give answers to tasks.

For a worker to be disqualified, the following criteria should be met:

The worker should have more than one retracted round.
Other workers should have more than 0 gold answers.
The worker's Gold Accuracy should be lower than the other workers' Gold Accuracy median*0.9.

The Qualification Score to be assigned to the worker after the disqualification can be specified in the Disqualification_score parameter.

(($rs.roundRetracted && $rac.retractedRounds + 1 > 1) || (!$rs.roundRetracted && $rac.retractedRounds > 1) )
and
eval($aggregatedWorkerStatistics.otherWorkersGoldAnswerCount > 0 &&
RuleContextUtils.getTaskWorkerGoldAccuracy($rac) < $aggregatedWorkerStatistics.otherWorkersGoldAccuracyMedian * 0.9)

Disqualify worker when response time is too high

For a worker to be disqualified, the following criteria should be met:

The worker should have more than one retracted round.
The worker task response median should be lower than the other workers' task response median/2.

The Qualification Score to be assigned to the worker after the disqualification can be specified in the Disqualification_score parameter.

(($rs.roundRetracted && $rac.retractedRounds + 1 > 1) || (!$rs.roundRetracted && $rac.retractedRounds > 1) )      and  eval($aggregatedWorkerStatistics.WorkerTaskResponseTimeMedian < $aggregatedWorkerStatistics.otherWorkersTaskResponseTimeMedian / 2)

Disqualify worker when performance is low

For a worker to be disqualified, the following criteria should be met:

The worker's total task number should be greater than one.
The number of retracted tasks for the worker in the database should be greater than total*50/100.

($aggregatedWorkerStatistics.totalTaskCount > 1)
and
(eval($rs.assignmentRetracted && $rac.retractedTasks + 1 > ($aggregatedWorkerStatistics.totalTaskCount * 50) / 100) or
eval(!$rs.assignmentRetracted && $rac.retractedTasks > ($aggregatedWorkerStatistics.totalTaskCount * 50) / 100) )

WorkFusion Documentation

10.1.2

Create Manual Task

Create Qualification Task

Design Manual Task

Add rules and rule templates

Select advanced options

Run Manual Task

Create Business Process

Design Business Process

Add Bot Task

Logging and tracking

Run AutoML Business Process

View results

Worker

Requester

Administrator

Tagging over Document

View results

Overview

Out-of-the-box dashboards

Custom dashboards

Adjudication Rule parameters

Majority value

Maximum assignments

Evaluation frequency

Normal accuracy % vs Super accuracy %

Normal Accuracy score vs Super Accuracy score

Retract Worker Answers

Disqualify workers

Retract worker when response time is too high

Retract round when Gold Accuracy is low

Retract round when Accuracy is low

Disqualify worker when Gold Accuracy is low

Disqualify worker when response time is too high

Disqualify worker when performance is low