Create Adjudication Rules
Select Campaigns→ Rules.
Select the rule with the template you need and copy it.
caution
Do not change the settings of the initial rule!
Select the rule you created. Name the rule so that you could possibly reuse it for other questions.
Set up the majority values and max assignments limits, depending on the use case you have.
The Majority value is the minimum number of workers required to complete the task so that the results are available in the output snapshot. The task is not completed unless the number of workers participating in the task reaches or exceeds the specified majority value.
Max # of Assignments is the total amount of workers allowed to participate in a task. For that reason, Max # of Assignments CANNOT be less than the Majority value.
For example, when Majority value (Min # of Assignments) = 4, Max # of Assignments = 6, it means that four worker answers are enough to find the majority. The 5th worker’s answer is utilized when the majority is not found. If the majority is found among five workers, no work is distributed beyond the five workers count.
Save the rule.
Select a Manual Task, go to the Run tab → Advanced Options. Choose the rule you have created, set up Min and Max # of Assignments, and click OK.
Run the task.
To exclude specific fields from adjudication, go to the Manual Task, select the Design tab, click the answer button to view Advanced options, and select Do not use in Adjudication (SA).
Proceed with Adjudication steps in case of six or more SMEs
| Number of SMEs | Adjudication rule |
|---|---|
| 1-5 SMEs | Start with one assignment 2+1 pay all, Start with one assignment 3+2 pay all |
| 6+ SMEs | Start with one assignment 2+1 pay all, Start with one assignment 3+2 pay all or create your own adjudication rule |
In case you have more than five SMEs, it is recommended to set up the 2+1 or 3+2 rules.
Before launching a task, make sure you set up appropriate minimum and maximum numbers of assignments (min and max number of workers required to complete the task).