Qualification Rules

Qualification Rules are executed instead of Adjudication Rules for Qualification Tasks. They are used to evaluate worker performance during Qualification runs. The Qualification Rules allow Intelligent Automation (IA) Cloud to grant Accuracy Based Qualifications to workers based on their performance in the course of such runs.

IA Cloud provides the Generic Qualification Rule by default. You can modify the rule or create one.

Qualification Rule

The first part of the rule serves to initialize parameters. The second part evaluates worker scores after they have taken a specified number of tasks. The number is defined in the TOTAL_HITS_LIMIT parameter of the rule.

Qualification Rule example

For example, after a worker has taken ten tasks, the second part of the rule begins to evaluate the worker's performance in the Qualification run. The following outcome options are possible:

If the worker scores higher than or equal to the value defined in the NORMAL_ACCURACY parameter and less than the value defined in the ELITE_ACCURACY parameter, the worker is granted the Accuracy Based Qualification with the score defined in the NORMAL_SCORE parameter.
If the worker scores higher than the value defined in the ELITE_ACCURACY parameter, the worker is granted the Accuracy Based Qualification with the score defined in the ELITE_SCORE parameter.

Some Qualification Rules also include BUSTLER_TIME_IN_SECONDS. The parameter detects bustlers—people who try to complete tasks without properly attempting them or reading instructions. If the response time for the tasks was less than or equal to the parameter, the Assignment is rejected and retracted from the worker.

Finally, the system checks the MAX_ASSIGNMENT_LIMIT parameter. If the number of assignments created for the task equals or exceeds the parameter values, the HIT is disposed of, meaning no further assignments are created for the run.

Qualification rule example

package com.freedomoss.requester;
#list any import classes here.
import com.freedomoss.objective.model.RuleContext;
import com.freedomoss.objective.model.RuleAssigmentContext;
import com.freedomoss.objective.model.RuleQuestionContext;
import org.slf4j.Logger;
import java.util.ArrayList;
import java.util.Map;
import java.lang.String;
#declare any global variables here
global RuleContext source
global Logger log
global Map params

rule "Rule context initialization"
auto-focus true
no-loop
agenda-group "initialization-group"
    when
        $ctx:RuleContext(initialized == false);
    then
        # set parameters
        #$ctx.addProperty(RuleContext.MAJORITY_TYPE, RuleContext.MajorityType.COUNT);
        #$ctx.addProperty(RuleContext.MAJORITY_VALUE, new Integer(2));
        #$ctx.addProperty(RuleContext.DISTINCT_ANSWERS, true);
        #$ctx.addProperty(RuleContext.SKIP_ANSWER + "toCheck", "");
        #$ctx.addProperty(RuleContext.ACCEPT_ANSWER + "toCheck", "b");
        #$ctx.addProperty(RuleContext.SKIP_ANSWER, "");
        #$ctx.addProperty(RuleContext.ACCEPT_ANSWER, "");
        # $ctx.addProperty(RuleContext.SKIP_EMPTY_ANSWERS, true);
         $ctx.addProperty(RuleContext.MAJORITY_TYPE, RuleContext.MajorityType.PERCENTAGE);
         $ctx.addProperty(RuleContext.MAJORITY_VALUE, new Double(0.5));
        # insert processed facts into memory
        $ctx.updateWorkingMemory();
        # move to business rules
        kcontext.getKnowledgeRuntime().getAgenda().getAgendaGroup("calculation").setFocus();
end
rule "000. Test Rule - Evaluate every hit"
agenda-group "calculation"
dialect "mvel"
salience 260
no-loop
 when
     $ctx:RuleContext(initialized == true)
     $rac:RuleAssigmentContext($campaignStatistic:campaignStatistic)
     #evaluate run average response time
     #eval($campaignStatistic.totalHits >= 30 && $campaignStatistic.accuracy >= 90)
 then
     $ctx.logExecutedRule("Accuracy-:" + $campaignStatistic.getAccuracy());
     $ctx.logExecutedRule("GAccuracy-:" + $campaignStatistic.getGoldAccuracy());
     $ctx.logExecutedRule("TotalHits-:" + $campaignStatistic.getTotalHits());
end
rule "001. Evaluate every 20 hits and grant Qualified to work on Spanish sentiment Tasks set score 70 "
agenda-group "calculation"
dialect "mvel"
salience 250
no-loop
 when
     $ctx:RuleContext(initialized == true)
     $rac:RuleAssigmentContext($campaignStatistic:campaignStatistic)
     #evaluate run average response time
     eval($campaignStatistic.totalHits >= 30 && $campaignStatistic.goldAccuracy < 0.8)
 then
     //getRunStatistic, getCampaignStatistic
     $rac.grandQualification(RuleContext.ACCURACY_BASED_QUALIFICATION, 70);
     $ctx.logExecutedRule(kcontext.getRule().getName());
     $ctx.logExecutedRule("GAccuracy-:" + $campaignStatistic.getGoldAccuracy());
end

rule "001. Evaluate every 20 hits and grant Qualified to work on Spanish sentiment Tasks set score 80"
agenda-group "calculation"
dialect "mvel"
salience 250
no-loop
 when
     $ctx:RuleContext(initialized == true)
     $rac:RuleAssigmentContext($campaignStatistic:campaignStatistic)
     #evaluate run average response time
     eval($campaignStatistic.totalHits >= 30 && $campaignStatistic.goldAccuracy >= 0.8 && $campaignStatistic.goldAccuracy < 0.9)
 then
     $rac.grandQualification(RuleContext.ACCURACY_BASED_QUALIFICATION, 80);
     $ctx.logExecutedRule(kcontext.getRule().getName());
     $ctx.logExecutedRule("GAccuracy-:" + $campaignStatistic.getGoldAccuracy());
end
rule "001. Evaluate every 20 hits and grant Qualified to work on Spanish sentiment Tasks set score 90"
agenda-group "calculation"
dialect "mvel"
salience 250
no-loop
 when
     $ctx:RuleContext(initialized == true)
     $rac:RuleAssigmentContext($campaignStatistic:campaignStatistic)
     #evaluate run average response time
     eval($campaignStatistic.totalHits >= 30 && $campaignStatistic.goldAccuracy >= 0.9)
 then
     $rac.grandQualification(RuleContext.ACCURACY_BASED_QUALIFICATION, 90);
     $ctx.logExecutedRule(kcontext.getRule().getName());
     $ctx.logExecutedRule("GAccuracy-:" + $campaignStatistic.getGoldAccuracy());
end
rule "3. Approve assignment"
agenda-group "calculation"
dialect "mvel"
salience 20
no-loop
    when
        $ctx:RuleContext(inAdvanceKnownValues:inAdvanceKnownValues, inAdvanceKnownValues.size != 0)
        $rac:RuleAssigmentContext()
        eval($rac.containAll($ctx.inAdvanceKnownValues(), 10))
    then
        $ctx.addApproved($rac);
        $ctx.logExecutedRule(kcontext.getRule().getName(), $rac);
end
rule "4. Reject assignment"
agenda-group "calculation"
dialect "mvel"
salience 20
no-loop
    when
        $ctx:RuleContext(inAdvanceKnownValues:inAdvanceKnownValues, inAdvanceKnownValues.size != 0);
        $rac:RuleAssigmentContext();
        eval(!$rac.containAll($ctx.inAdvanceKnownValues(), 90))
    then
        $ctx.addRejected($rac);
        $ctx.logExecutedRule(kcontext.getRule().getName(), $rac);
end

rule "5. Extend HIT"
agenda-group "calculation"
dialect "mvel"
salience 10
no-loop
    when
        $ctx:RuleContext(assigments.size() < 8);
    then
        $ctx.setExtendHit(true);
        $ctx.logExecutedRule(kcontext.getRule().getName());
end
rule "6. Dispose HIT"
agenda-group "calculation"
dialect "mvel"
salience 10
no-loop
    when
        $rac:RuleAssigmentContext();
        $ctx:RuleContext(assigments.size() >= 8)
    then
        $ctx.addApproved($rac);
        //$ctx.addRejected($rac);
        $ctx.logExecutedRule(kcontext.getRule().getName());
end

Qualification rule parameters

Parameter	Parameter	Description
Maximum Assignments		Determines the maximum number of assignments to be created for a Qualification run. If the required number of qualified workers is not reached and the task needs to be extended by posting more assignments, IA Cloud will not exceed the value specified in the parameter when posting more assignments.
Normal Accuracy %	Elite Accuracy %	IA Cloud allows assigning two rankings to workers—Normal and Elite. The two values make it possible to determine what is Normal and what is Elite. You can set the percentage of correct answers required from a worker in responses to Gold Data tasks to achieve either the Normal or Elite rankings. For example, if you set Normal Accuracy to 75%, the worker is given a Normal ranking if their answers in Gold tasks are at least 75% correct. The parameter is sometimes referred to as `UPPER_LIMIT` and `LOWER_LIMIT`.
Normal accuracy score	Elite accuracy score	On some endpoints, like Mechanical Turk, a Qualification needs to have a score associated with it. The score is different from the Gold Accuracy score, which is the percentage of the worker's correct answers in Gold Data tasks. So, based on the Gold Accuracy score, the worker is assigned a ranking as described above. Then, the worker is assigned a score based on the two parameters. Basically, these parameters determine the score granted to workers based on the ranking they achieved. For example, if a worker was ranked as Elite, and the Elite Accuracy Score parameter is set to 90, the worker is scored 90 for the Qualification associated with the task.
Bustler Time		Bustlers are workers who do not spend enough time on a task. This often is an indication that their answers are of poor quality and should not be accepted. The parameter allows you to determine how much time a worker needs to spend on each task not to be labeled a bustler.

WorkFusion Documentation

9.2

Create Manual Task

Create Qualification Task

Design Manual Task

Add rules and rule templates

Select advanced options

Run Manual Task

Create Business Process

Design Business Process

Add Bot Task

Logging and tracking

Run AutoML Business Process

View results

Worker

Requester

Administrator

Tagging over Document

View results

Environment setup

Initiation and planning

General implementation

ML implementation

RPA implementation

Post-implementation

Qualification Rule example

Qualification rule parameters