Adjudication
Adjudication is a process that provides the logic on how correct answers are derived, what determines the accuracy of the work, and whether it should be accepted once completed. Adjudication rules use a majority rule algorithm that you can set to determine how many Workers you want to have to agree in order to reach a consensus.
Adjudication in the WorkFusion platform is a process that ensures the quality of work performed by crowd Workers. There are several such Rules which are created by default on your instance of the platform. In addition, you can also create your own rules.
Every Task that you create has an Adjudication Rule associated with it. You can view or edit the Rule associated with your Task in the Advanced Options of the Run Task tab.
| Field | Description |
|---|---|
| Min # of Assignments | The minimum number of Workers required to complete each Worker Task. The value set determines the Minimum number of workers REQUIRED to complete a record before it is considered completed. This means you will NOT see results from this record in your snapshot until the minimum number of assignments is met. Example: if you want 80 different Workers to complete the same Worker Task (for example, questionnaire), create a Task with one Record as input data and set the "Min # of Assignments" = 80. |
| Max # of Assignments | If the Task reaches this number of Workers, no new Worker Tasks will be issued. |
| Adjudication Rule | Select the Adjudication Rule with the logic schema that best suits your Task. You can select one of the included Adjudication Rules from the drop-down menu or create your own one. |
| Price Strategy | This strategy is intended to update Task price automatically (for example, every day). It is based on specific rules with Price type. See the Price Rules topic. |
| Enable Manual Review | If you wish to qualify Workers without Gold Data or other Workers, review their work manually. |
To understand the Adjudication Rules concept, see the following description of two Rules loaded by default:
Start with 1 assignment 2+1 rule (3 assignments max, pay all) This rule means that we need 2 identical answers to reach the majority. If their answers are different from each other, then utilize a third Worker to establish a majority based answer. Once that majority is reached, pay all of them. If there is still no match, then since the Max # of Assignments field was set to 3, no further Worker Tasks are posted. The output file (snapshot) will indicate the answers provided by the 3 workers and that Confidence is "1 out of 3".
Start with 1 assignment 3+2 rule (5 assignments max, pay all) This rule means that three Workers are set to work on this Worker Task. If their answers are different from each other, then utilize an additional two Workers to establish a majority based answer. Once that majority is reached, pay all of them.
Edit Rule
To set Rule parameters:
Click the Edit Rule link to the right of the Adjudication Rule dropdown. Depending on the Adjudication Rule selected, there will be a specific set of Rule Variable inputs (i.e. Check every i-th gold question, Majority value, Worst accuracy level upper limit).
Select the Adjudication Strategy.
Set the Rule Variables.
Click the Save button.
note
The Rule you are saving might be used in other Tasks.
You can also edit the Rule code by clicking the Switch to Advanced View link. To get detailed info, see the Rules and Rule Templates topic.
You can optionally edit the Price Strategy the same way as the Adjudication Rule.
