Adjudication Rules
Adjudication is a process that ensures the quality of answers provided by Workers. These Rules allow you to configure this Adjudication process.
There are several such rules which are created by default on your instance of the platform. These can be accessed from the Rules page. In addition, you can also create your own rules.
Every Human Task that you create has an Adjudication Rule associated with it. You can see the rule associated with your Task and even change the rule by going to the Run Task > Advanced Options > Adjudication.
The Adjudication Rule Configuration in the screenshot above tells the Intelligent Automation Cloud to use a minimum of 2 Assignments and a maximum of 3 Assignments and to use the "2 + 1, pay all" Rule for this particular Task.
This means that the platform will initially post two assignments of the Task on the endpoint (for example WorkSpace or Amazon Mechanical Turk). If there are multiple Records in the input file, it will post two assignments for each of these records. Then as the Workers complete these assignments, the platform will check if the answers match or not between the two assignments. If both match then the Task is accepted and no further assignments are posted. If they do not match, then a third assignment is posted. When this third assignment is submitted, the platform compares this result with the previous two. If it finds a match then it accepts the answers that were matching. If there is still no match, then since the Max # of Assignments field was set to 3, no further assignments are posted. The output file (snapshot) will contain the answers provided by the 3 Workers and that no confidence was found.
Adjudication Rules can be configured by Rule Parameters in the Basic View and the actual Rules Code in the Advanced View.
Adjudication Rule Example
package com.freedomoss.requester;
#list any import classes here.
import com.freedomoss.objective.model.RuleContext;
import com.freedomoss.objective.model.RuleContext.MajorityType;
import com.freedomoss.objective.model.RuleAssigmentContext;
import com.freedomoss.objective.model.RuleQuestionContext;
import com.freedomoss.requester.model.AwsHitQuestion;
import com.freedomoss.requester.model.AwsHitQuestionItem;
import com.freedomoss.requester.model.AwsHitQuestionAnswerItem;
import org.slf4j.Logger;
import java.util.ArrayList;
import java.util.List;
import java.util.Set;
import java.util.Map;
import java.util.Iterator;
#declare any global variables here
global RuleContext source
global Logger log
global Map params
rule "Rule context initialization"
auto-focus true
no-loop
dialect "mvel"
agenda-group "initialization-group"
when
$ctx:RuleContext(initialized == false);
then
# set parameters
$ctx.properties[RuleContext.MAJORITY_TYPE] = RuleContext.MajorityType.COUNT;
$ctx.properties[RuleContext.MAJORITY_VALUE] = new Integer(2);
$ctx.properties[RuleContext.MAX_ASSIGNMENT_LIMIT] = new Integer(3);
$ctx.properties[RuleContext.MAJORITY_HIT_THRESHOLD] = new Double(100/100);
$ctx.properties[RuleContext.ASSIGNMENT_APPROVE_THRESHOLD] = new Double(0.5);
# insert processed facts into memory
$ctx.updateWorkingMemory();
# move to business rules
kcontext.getKnowledgeRuntime().getAgenda().getAgendaGroup("calculation").setFocus();
$ctx.logExecutedRule(kcontext.getRule().getName());
end
rule "Worst accuracy rule - Evaluate every 5 gold question and set Accuracy based qualification score to 70 if gold accuracy goes up < 75 percents"
agenda-group "calculation"
dialect "mvel"
salience 250
no-loop
when
$ctx:RuleContext(initialized == true)
$rac:RuleAssigmentContext($campaignStatistic:campaignStatistic, runStatistic.totalGoldQuestions > 0)
#evaluate run average response time
eval($campaignStatistic.totalGoldQuestions > 0 &&
($campaignStatistic.totalGoldQuestions % 5) == 0 && $campaignStatistic.goldAccuracy < (75/100))
then
$rac.grandQualification(RuleContext.ACCURACY_BASED_QUALIFICATION, 70);
$ctx.logExecutedRule(kcontext.getRule().getName());
end
rule "Normal accuracy rule - Evaluate every 5 gold question and set Accuracy based qualification score to 80 if gold accuracy goes up >= 75 percents and < 90"
agenda-group "calculation"
dialect "mvel"
salience 250
no-loop
when
$ctx:RuleContext(initialized == true)
$rac:RuleAssigmentContext($campaignStatistic:campaignStatistic, runStatistic.totalGoldQuestions > 0)
#evaluate run average response time
eval($campaignStatistic.totalGoldQuestions > 0 &&
($campaignStatistic.totalGoldQuestions % 5) == 0 &&
$campaignStatistic.goldAccuracy >= (75/ 100) && $campaignStatistic.goldAccuracy < (90 / 100))
then
$rac.grandQualification(RuleContext.ACCURACY_BASED_QUALIFICATION, 80);
$ctx.logExecutedRule(kcontext.getRule().getName());
end
rule "Super accuracy rule - Evaluate every 5 gold question and set Accuracy based qualification score to 95 if gold accuracy goes up >= 90 percents"
agenda-group "calculation"
dialect "mvel"
salience 250
no-loop
when
$ctx:RuleContext(initialized == true)
$rac:RuleAssigmentContext($campaignStatistic:campaignStatistic, runStatistic.totalGoldQuestions > 0)
#evaluate run average response time
eval($campaignStatistic.totalGoldQuestions > 0 &&
($campaignStatistic.totalGoldQuestions % 5) == 0 &&
$campaignStatistic.goldAccuracy >= (90 / 100) )
then
$rac.grandQualification(RuleContext.ACCURACY_BASED_QUALIFICATION, 95);
$ctx.logExecutedRule(kcontext.getRule().getName());
end
rule "0. Check, if exist majority"
agenda-group "calculation"
dialect "mvel"
salience 100
no-loop
when
$ctx:RuleContext (initialized == true, $gold:gold, $threshold:properties.MAJORITY_HIT_THRESHOLD)
eval($ctx.majorityWithoutGold().size() >= ($ctx.questions.size() - $gold.size()) * $threshold)
then
insert(new String("MAJORITY_FOUND"));
$ctx.logExecutedRule(kcontext.getRule().getName());
end
rule "1. Approve question by majority"
agenda-group "calculation"
dialect "mvel"
salience 90
no-loop
when
((String(toString == "MAJORITY_FOUND") and $ctx:RuleContext (initialized == true))
or
(not String(toString == "MAJORITY_FOUND") and $ctx:RuleContext (initialized == true, assignments.size == properties.MAX_ASSIGNMENT_LIMIT)))
$rqc:RuleQuestionContext()
then
$rqc.approveQuestion($ctx.majority());
$ctx.logExecutedRule(kcontext.getRule().getName(), $rqc);
end
rule "3. Approve assignment (always)"
agenda-group "calculation"
dialect "mvel"
salience 50
no-loop
when
$ctx:RuleContext(initialized == true)
$rac:RuleAssigmentContext()
then
insert(new String("ASSIGNMENT_PROCESSED"));
$ctx.addApproved($rac);
$ctx.logExecutedRule(kcontext.getRule().getName(), $rac);
end
rule "5. Extend HIT"
agenda-group "calculation"
dialect "mvel"
salience 50
no-loop
when
not String(toString == "MAJORITY_FOUND")
$ctx:RuleContext(initialized == true, assignments.size < properties.MAX_ASSIGNMENT_LIMIT);
then
$ctx.setExtendHit(true);
$ctx.logExecutedRule(kcontext.getRule().getName());
end
rule "6. Dispose HIT, no majority"
agenda-group "calculation"
dialect "mvel"
salience 30
no-loop
when
not String(toString == "MAJORITY_FOUND");
$ctx:RuleContext(initialized == true, assignments.size >= properties.MAX_ASSIGNMENT_LIMIT);
$rac:RuleAssigmentContext();
then
$ctx.addApproved($rac);
$ctx.setDisposeHit(true);
$ctx.logExecutedRule(kcontext.getRule().getName());
end
Adjudication Rule Parameters
Majority Value
Determines how many Workers need to agree in their answers for the Assignment to be accepted and the Task to be closed. So for example if this parameter is set at 2, the Intelligent Automation Cloud will create 2 Assignments (instances) of the Task on the endpoint (WorkSpace, Mechanical Turk, etc.). If Workers submit the same answers for both Assignments then the Task is closed, meaning it is not posted again and the Workers are paid. But if it return different answers, then one more assignment is created. WorkFusion will continue posting Assignments until 2 Workers provide the same answer.
Increasing this value will ensure further quality of your work but it may result in more assignments being created and thus more Workers needing to be paid.
Maximum Assignments
Determines the maximum number of assignments that will be created during Adjudication. If Worker do not provide the needed majority and the Task needs to be extended by posting more assignments, then the Intelligent Automation Cloud will not exceed the value specified in this parameter when posting more assignments.
For example if the Majority Value is 2, and more assignments need to be posted to reach that majority, and the Maximum Assignments is 4, then at most 4 assignments of that Task will be posted. If after 4 assignments, the majority still has not been reached then the Task will be closed.
Increasing this value will ensure further quality of your work but it may result in more assignments being created and thus more Workers needing to be paid.
Evaluation Frequency
The Rule is also known as the CHECK_EVERY Rule. When performing Ongoing Qualifications, it determines after how many Gold Tasks a Worker performs to evaluate that Worker's score on this Task.
For example, if this parameter is set to 5 then after every 5 Gold Tasks, the system will evaluate Worker's performance and update Worker score.
Normal Accuracy % vs Super Accuracy %
The Intelligent Automation Cloud currently has two rankings that it assigns to Workers, Normal and Super. Normal Workers for a given Task usually are those who are known to perform it adequately well. Super Workers usually are those who are known to perform the given Task exceptionally well. These two parameters allow you to make the determination of what constitutes as "Normal" and "Super". They do this by allowing you to decide the percentage of correct answers that need to be present from the Worker in responses to Gold Data Tasks to achieve either the Normal or Super rankings.
For example if you set Normal Accuracy to 75% then if the Worker answers at least 75% correct on Gold Tasks they will be given a Normal Ranking. And if you set the Super Accuracy to 90% then the system will only grant them Super Accuracy if they respond correctly to at least 90% of the Gold Tasks presented to them. This parameter is sometimes referred to as "UPPER_LIMIT" and "LOWER_LIMIT".
Normal accuracy score vs Super accuracy score
On some endpoints, like Mechanical Turk, a Qualification needs to have a score associated with it. This score is different from the Gold Accuracy score which is the percentage the Worker answered correctly on the Gold Data Tasks. So based on the Gold Accuracy Score, the Worker is assigned a ranking as described above. Then based on this ranking, the Worker is assigned a score based on these two parameters. In essence these parameters determine the score granted to the Workers based on the Ranking they achieved. So for example if they achieved a ranking of Super and the Super Accuracy Score parameter is set to 90 then they will be assigned a score of 90 on the Qualification associated with this Task.
Adjudication Rules can contain other Rules that verify Worker statistics and execute some actions based on the results. See the rules below.
Retracting Worker Answers
The functionality is intended to exclude answers from final result (for example, if Worker is a cheater). If a Task contains retracted answers, the Majority and Confidence will be recalculated. Retract button is available only for human not qualification Tasks.
Answers can be retracted:
- Manually (Task > View Results > Workers tab > Retract Worker Answers). Worker can continue to work on tasks in this run. Answers that have been given by this Worker after manual retract won't be retracted automatically.
- By the Adjudication Rule. If answers have been retracted by the rule, future answers of the Worker in this run will also be retracted and rejected. Retracted assignment won't be included in Max assignment limit counting. If task contains more than 1 assignment, additional assignment will be created instead of retracted one.
Retracting rule example
rule "Retract less 7 sec"
agenda-group "calculation"
dialect "mvel"
salience 20
no-loop
when
$ctx:RuleContext();
$rac:RuleAssigmentContext();
eval($rac.assignment.submitTime.time - $rac.assignment.acceptTime.time <= 7000);
then
$ctx.retractWorker($rac.assignment.WorkerNativeId);
$ctx.logExecutedRule(kcontext.getRule().getName());
end
rule "Retract already retracted"
agenda-group "calculation"
dialect "mvel"
salience 20
no-loop
when
$ctx:RuleContext();
$rac:RuleAssigmentContext(WorkerRetracted == true);
then
$ctx.retractWorker($rac.assignment.WorkerNativeId);
$ctx.logExecutedRule(kcontext.getRule().getName());
end
rule "Retract low confidence Worker"
agenda-group "calculation"
dialect "mvel"
salience 20
no-loop
when
$ctx:RuleContext();
$confidencesForAnswer: java.util.Map() from $ctx.WorkerConfidences.values();
$entry: Map.Entry(value < 0.7) from $confidencesForAnswer.entrySet();
then
$ctx.retractWorker($entry.key);
$ctx.logExecutedRule(kcontext.getRule().getName());
end
rule "Extend HIT when all assignments retracted"
agenda-group "calculation"
dialect "mvel"
salience 50
no-loop
when
$ctx:RuleContext(initialized == true);
eval($ctx.totalAssignmentCount < $ctx.properties.MAX_ASSIGNMENT_LIMIT);
eval($ctx.assignments.empty);
then
$ctx.setExtendHit(true);
$ctx.logExecutedRule(kcontext.getRule().getName());
end
Disqualifying Workers
This rule allows to disqualify Workers if their Performance, Accuracy, or Gold Accuracy does not meet the specified requirements.
Disqualify rule example
package com.freedomoss.requester;
#list any import classes here.
import com.freedomoss.objective.model.RuleContext;
import com.freedomoss.objective.model.RuleContext.MajorityType;
import com.freedomoss.objective.model.RuleAssigmentContext;
import com.freedomoss.objective.model.RuleQuestionContext;
import com.freedomoss.requester.model.AwsHitQuestion;
import com.freedomoss.requester.model.AwsHitQuestionItem;
import com.freedomoss.requester.model.AwsHitQuestionAnswerItem;
import com.freedomoss.objective.model.RuleContextUtils;
import com.freedomoss.objective.facts.RetractState;
import org.slf4j.Logger;
import java.util.ArrayList;
import java.util.List;
import java.util.Set;
import java.util.Map;
import java.util.Iterator;
#declare any global variables here
global RuleContext source
global Logger log
global Map params
rule "Rule context initialization"
auto-focus true
no-loop
dialect "mvel"
agenda-group "initialization-group"
when
$ctx:RuleContext(initialized == false);
then
# set parameters
$ctx.properties[RuleContext.MAJORITY_TYPE] = RuleContext.MajorityType.COUNT;
$ctx.properties[RuleContext.MAJORITY_VALUE] = new Integer(2);
$ctx.properties[RuleContext.MAX_ASSIGNMENT_LIMIT] = new Integer(3);
$ctx.properties[RuleContext.MAJORITY_HIT_THRESHOLD] = new Double(100/100);
$ctx.properties[RuleContext.ASSIGNMENT_APPROVE_THRESHOLD] = new Double(0.5);
# insert processed facts into memory
$ctx.updateWorkingMemory();
# move to business rules
kcontext.getKnowledgeRuntime().getAgenda().getAgendaGroup("calculation").setFocus();
$ctx.logExecutedRule(kcontext.getRule().getName());
end
rule "Worst accuracy rule - Evaluate every 5 gold question and set Accuracy based qualification score to 70 if gold accuracy goes up < 75 percents"
agenda-group "calculation"
dialect "mvel"
salience 250
no-loop
when
$ctx:RuleContext(initialized == true)
$rac:RuleAssigmentContext($campaignStatistic:campaignStatistic)
#evaluate run average response time
eval($campaignStatistic.totalGoldQuestions > 0 &&
($campaignStatistic.totalGoldQuestions % 5) == 0 && $campaignStatistic.goldAccuracy < (75/100))
then
$rac.grantQualification(RuleContext.ACCURACY_BASED_QUALIFICATION, 70);
$ctx.logExecutedRule(kcontext.getRule().getName());
end
rule "Normal accuracy rule - Evaluate every 5 gold question and set Accuracy based qualification score to 80 if gold accuracy goes up >= 75 percents and < 90"
agenda-group "calculation"
dialect "mvel"
salience 250
no-loop
when
$ctx:RuleContext(initialized == true)
$rac:RuleAssigmentContext($campaignStatistic:campaignStatistic)
#evaluate run average response time
eval($campaignStatistic.totalGoldQuestions > 0 &&
($campaignStatistic.totalGoldQuestions % 5) == 0 &&
$campaignStatistic.goldAccuracy >= (75/ 100) && $campaignStatistic.goldAccuracy < (90 / 100))
then
$rac.grantQualification(RuleContext.ACCURACY_BASED_QUALIFICATION, 80);
$ctx.logExecutedRule(kcontext.getRule().getName());
end
rule "Super accuracy rule - Evaluate every 5 gold question and set Accuracy based qualification score to 95 if gold accuracy goes up >= 90 percents"
agenda-group "calculation"
dialect "mvel"
salience 250
no-loop
when
$ctx:RuleContext(initialized == true)
$rac:RuleAssigmentContext($campaignStatistic:campaignStatistic)
#evaluate run average response time
eval($campaignStatistic.totalGoldQuestions > 0 &&
($campaignStatistic.totalGoldQuestions % 5) == 0 &&
$campaignStatistic.goldAccuracy >= (90 / 100) )
then
$rac.grantQualification(RuleContext.ACCURACY_BASED_QUALIFICATION, 95);
$ctx.logExecutedRule(kcontext.getRule().getName());
end
rule "0. Check, if exist majority"
agenda-group "calculation"
dialect "mvel"
salience 100
no-loop
when
$ctx:RuleContext (initialized == true, $gold:gold, $threshold:properties.MAJORITY_HIT_THRESHOLD)
eval($ctx.assignments.size() > 0 && $ctx.majorityWithoutGold().size() >= ($ctx.questions.size() - $gold.size()) * $threshold)
then
insert(new String("MAJORITY_FOUND"));
$ctx.logExecutedRule(kcontext.getRule().getName());
end
rule "1. Approve question by majority"
agenda-group "calculation"
dialect "mvel"
salience 90
no-loop
when
((String(toString == "MAJORITY_FOUND") and $ctx:RuleContext (initialized == true))
or
(not String(toString == "MAJORITY_FOUND") and $ctx:RuleContext (initialized == true, assignments.size == properties.MAX_ASSIGNMENT_LIMIT)))
$rqc:RuleQuestionContext()
then
$rqc.approveQuestion($ctx.majority());
$ctx.logExecutedRule(kcontext.getRule().getName(), $rqc);
end
rule "3. Approve assignment (always)"
agenda-group "calculation"
dialect "mvel"
salience 50
no-loop
when
$ctx:RuleContext(initialized == true)
$rac:RuleAssigmentContext()
then
insert(new String("ASSIGNMENT_PROCESSED"));
$ctx.addApproved($rac);
$ctx.logExecutedRule(kcontext.getRule().getName(), $rac);
end
rule "5. Extend HIT"
agenda-group "calculation"
dialect "mvel"
salience 50
no-loop
when
not String(toString == "MAJORITY_FOUND")
$ctx:RuleContext(initialized == true, assignments.size < properties.MAX_ASSIGNMENT_LIMIT);
then
$ctx.setExtendHit(true);
$ctx.logExecutedRule(kcontext.getRule().getName());
end
rule "6. Dispose HIT, no majority"
agenda-group "calculation"
dialect "mvel"
salience 30
no-loop
when
not String(toString == "MAJORITY_FOUND");
$ctx:RuleContext(initialized == true, assignments.size >= properties.MAX_ASSIGNMENT_LIMIT);
$rac:RuleAssigmentContext();
then
$ctx.addApproved($rac);
$ctx.setDisposeHit(true);
$ctx.logExecutedRule(kcontext.getRule().getName());
end
rule "Init retract states"
agenda-group "calculation"
dialect "mvel"
no-loop
when
$ctx:RuleContext(initialized == true)
$rac:RuleAssigmentContext()
then
insert(new RetractState($rac.assignmentId), true);
end
rule "7. Retract Worker when response time is too high"
agenda-group "calculation"
dialect "mvel"
salience 20
no-loop
when
$ctx:RuleContext(initialized == true)
$rac:RuleAssigmentContext($aggregatedWorkerStatistics:aggregatedWorkerStatistics)
$rs:RetractState(assignmentId == $rac.assignmentId, assignmentRetracted == false)
eval($aggregatedWorkerStatistics.otherWorkersAnswerCount > 1 &&
$rac.responseTime < $aggregatedWorkerStatistics.otherWorkersTaskResponseTimeMedian / 2)
then
log.info("Retract assignment for Worker with high response time: " + $rac.nativeWorkerId);
$ctx.retractAssignment($rac.nativeWorkerId);
modify($rs) {assignmentRetracted = true};
$ctx.logExecutedRule(kcontext.getRule().getName());
end
rule "8. Retract round when gold accuracy is low"
agenda-group "calculation"
dialect "mvel"
salience 19
no-loop
when
$ctx:RuleContext(initialized == true, $gold:gold)
$rac:RuleAssigmentContext($runStatistic:runStatistic)
$rs:RetractState(assignmentId == $rac.assignmentId, roundRetracted == false)
eval($ctx.completeHitCount > 0 && ($ctx.completeHitCount % 2) == 0 &&
($gold.size() > 0 || $runStatistic.totalGold > 0 ) && RuleContextUtils.getTaskWorkerGoldAccuracy($rac) * 100 < 90)
then
log.info("Retract round for Worker with low gold accuracy: " + $rac.nativeWorkerId);
$ctx.retractRound($rac.nativeWorkerId);
modify($rs) {roundRetracted = true};
$ctx.logExecutedRule(kcontext.getRule().getName());
end
rule "9. Retract round when accuracy is low"
agenda-group "calculation"
dialect "mvel"
salience 18
no-loop
when
$ctx:RuleContext(initialized == true)
$rac:RuleAssigmentContext($runStatistic:runStatistic)
$rs:RetractState(assignmentId == $rac.assignmentId, roundRetracted == false)
eval($ctx.completeHitCount > 0 && ($ctx.completeHitCount % 2) == 0 &&
$runStatistic.totalMajorityCount > 0 && RuleContextUtils.getTaskWorkerAccuracy($rac) * 100 < 90)
then
log.info("Retract round for Worker with low accuracy: " + $rac.nativeWorkerId);
$ctx.retractRound($rac.nativeWorkerId);
modify($rs) {roundRetracted = true};
$ctx.logExecutedRule(kcontext.getRule().getName());
end
rule "10. Disqualify when when gold accuracy is low"
agenda-group "calculation"
dialect "mvel"
salience 17
no-loop
when
$ctx:RuleContext(initialized == true, $gold:gold)
$rac:RuleAssigmentContext($aggregatedWorkerStatistics:aggregatedWorkerStatistics)
$rs:RetractState(assignmentId == $rac.assignmentId)
eval(($rs.roundRetracted && $rac.retractedRounds + 1 > 1) || (!$rs.roundRetracted && $rac.retractedRounds > 1) )
and
eval($aggregatedWorkerStatistics.otherWorkersGoldAnswerCount > 0 &&
RuleContextUtils.getTaskWorkerGoldAccuracy($rac) < $aggregatedWorkerStatistics.otherWorkersGoldAccuracyMedian * 0.9)
then
disqualifyWorker($rac, 65, log);
$ctx.logExecutedRule(kcontext.getRule().getName());
end
rule "11. Disqualify when response time is too high"
agenda-group "calculation"
dialect "mvel"
salience 16
no-loop
when
$ctx:RuleContext(initialized == true)
$rac:RuleAssigmentContext($aggregatedWorkerStatistics:aggregatedWorkerStatistics)
$rs:RetractState(assignmentId == $rac.assignmentId)
eval(($rs.roundRetracted && $rac.retractedRounds + 1 > 1) || (!$rs.roundRetracted && $rac.retractedRounds > 1) )
and
eval($aggregatedWorkerStatistics.WorkerTaskResponseTimeMedian < $aggregatedWorkerStatistics.otherWorkersTaskResponseTimeMedian / 2)
then
disqualifyWorker($rac, 65, log);
$ctx.logExecutedRule(kcontext.getRule().getName());
end
rule "12. Disqualify when performance is low"
agenda-group "calculation"
dialect "mvel"
salience 15
no-loop
when
$ctx:RuleContext(initialized == true)
$rac:RuleAssigmentContext($aggregatedWorkerStatistics:aggregatedWorkerStatistics)
$rs:RetractState(assignmentId == $rac.assignmentId)
eval($aggregatedWorkerStatistics.totalTaskCount > 1)
and
(eval($rs.assignmentRetracted && $rac.retractedTasks + 1 > ($aggregatedWorkerStatistics.totalTaskCount * 50) / 100) or
eval(!$rs.assignmentRetracted && $rac.retractedTasks > ($aggregatedWorkerStatistics.totalTaskCount * 50) / 100) )
then
disqualifyWorker($rac, 65, log);
$ctx.logExecutedRule(kcontext.getRule().getName());
end
function void disqualifyWorker(RuleAssigmentContext rac, int score, Logger log) {
RuleContext rc = rac.getParent();
log.info("Disqualifying Worker: " + rac.getNativeWorkerId());
String autoGrantedQualificationUUID = RuleContextUtils.getAutoGrantedQualificationUuid(rc);
if (autoGrantedQualificationUUID != null) {
log.info("Changing auto granted qualification " + autoGrantedQualificationUUID + " score to " + score);
rac.forceGrantQualification(autoGrantedQualificationUUID, score);
} else {
String accuracyQualificationUUID = RuleContextUtils.getAccuracyBasedQualificationUuid(rc);
if(accuracyQualificationUUID != null) {
log.info("Changing accuracy based qualification " + accuracyQualificationUUID + " score to " + score);
rac.forceGrantQualification(accuracyQualificationUUID, score);
}
}
}
Retract Worker when Response Time is too high
This part marks assignments as Retracted in DB if Worker answers very fast. To mark an assignment:
- Should be more than 1 answer from other Workers (number of answers can be specified in Time_Check_threshold parameter)
- Time spend on task should be lower than (Time Median of other Workers/2). All answers greater than 90th percentile will be excluded from median counting.
($aggregatedWorkerStatistics.otherWorkersAnswerCount \> 1 &&
$rac.responseTime \< $aggregatedWorkerStatistics.otherWorkersTaskResponseTimeMedian / 2)
Retract Round when Gold Accuracy is low
This part retracts round for the Worker. If Worker has answers that were marked as retracted in DB - such answers will be retracted on UI and HIT will be extended according to the rule.
To retract round in this case:
- Task should have more than 0 completed HITs
- Retract_Round_Size parameter should be reached
- Total Golds in run should be greater than 0
- Worker GA*100 should be lower than 90 (90 - parameter that can be specified in Gold_Accuracy_Level_Limit for retract)
($ctx.completeHitCount > 0 && ($ctx.completeHitCount % 2) == 0 &&
($gold.size() > 0 || $runStatistic.totalGold > 0 ) && RuleContextUtils.getTaskWorkerGoldAccuracy($rac) * 100 < 90)
Retract Round when Accuracy is low
This part retracts round when Worker's accuracy is low.
To retract round:
- Task should have more than 0 completed HITs
- Retract_Round_Size parameter should be reached
- Total majority count should be greater than 0
- Worker Accuracy*100 should be lower than 90 (90 - parameter that can be specified in Accuracy_Level_Limit for retract)
($ctx.completeHitCount > 0 && ($ctx.completeHitCount % 2) == 0 &&
$runStatistic.totalMajorityCount > 0 && RuleContextUtils.getTaskWorkerAccuracy($rac) * 100 < 90)
Disqualify Worker when when Gold Accuracy is low
This part of the rule disqualifies Worker - lowers Worker's qualification (so Worker cannot continue to answer on tasks).
To disqualify:
- Worker should have more than 1 retracted round
- Other Workers should have gold answers (more than 0)
- Worker Gold Accuracy should be lower than other Workers gold accuracy median * 0.9
Qualification Score that Worker will have after disqualification can be specified in Disqualification_score parameter
(($rs.roundRetracted && $rac.retractedRounds + 1 > 1) || (!$rs.roundRetracted && $rac.retractedRounds > 1) )
and
eval($aggregatedWorkerStatistics.otherWorkersGoldAnswerCount > 0 &&
RuleContextUtils.getTaskWorkerGoldAccuracy($rac) < $aggregatedWorkerStatistics.otherWorkersGoldAccuracyMedian * 0.9)
Disqualify Worker when Response Time is too high
To disqualify:
- Worker should have more than 1 retracted round.
- Worker task response median should be lower than other Workers tasks response median / 2.
Qualification Score that Worker will have after disqualification can be specified in Disqualification_score parameter.
(($rs.roundRetracted && $rac.retractedRounds + 1 > 1) || (!$rs.roundRetracted && $rac.retractedRounds > 1) ) and eval($aggregatedWorkerStatistics.WorkerTaskResponseTimeMedian < $aggregatedWorkerStatistics.otherWorkersTaskResponseTimeMedian / 2)
Disqualify Worker when Performance is low
To disqualify:
- Worker's total tasks number should be greater than 1.
- Number of retracted tasks for Worker in DB should be greater than total*50/100.
($aggregatedWorkerStatistics.totalTaskCount > 1)
and
(eval($rs.assignmentRetracted && $rac.retractedTasks + 1 > ($aggregatedWorkerStatistics.totalTaskCount * 50) / 100) or
eval(!$rs.assignmentRetracted && $rac.retractedTasks > ($aggregatedWorkerStatistics.totalTaskCount * 50) / 100) )