Adjudication
Adjudication is a process that provides the logic on how correct answers are derived, what determines the accuracy of the work, and whether it should be accepted once completed. Adjudication rules use the majority rule algorithm you can set to determine how many workers you want to agree to reach a consensus.
In the WorkFusion platform, adjudication is a process that ensures the quality of the work performed by Crowd workers. On a platform instance, there are several such rules created by default. In addition, you can also create your own rules.
Every task you create has an Adjudication Rule associated with it. To view or edit the rule associated with your task, go to Run tab > Advanced Options.
| Field | Description |
|---|---|
| Min # of Assignments | The minimum number of workers required to complete each task. The value set determines the minimum number of workers REQUIRED to complete a record before it is considered completed. This means you can NOT see the results from the record in your snapshot until the minimum number of assignments is met. For example, if you want 80 different workers to complete the same task (for example, a questionnaire), create a task with one Record as input data and set the Min # of Assignments to 80. |
| Max # of Assignments | If the task reaches the number of workers, no new worker tasks are issued. |
| Adjudication Rule | Select the Adjudication Rule with the logic schema that best suits your task. You can select one of the included Adjudication Rules from the drop-down menu or create your own. |
| Price Strategy | The strategy is intended to update the task price automatically,for example, every day). It is based on specific rules with Price type. See the Price Rules topic. |
| Enable Manual Review | If you wish to qualify workers without Gold Data or other workers, review their work manually. |
To understand the Adjudication Rule concept, see the following description of the two rules loaded by default:
Start with 1 assignment 2+1 rule (3 assignments max, pay all): this rule means that you need two identical answers to reach the majority. If their answers are different from each other, utilize a third worker to establish a majority based on the answer. Once that majority is reached, pay all of them. If there is still no match, no further worker tasks are posted since the Max # of Assignments field was set to three. The output file (snapshot) indicates the answers provided by the three workers and that the Confidence is 1 out of 3.
Start with 1 assignment 3+2 rule (5 assignments max, pay all): this rule means that three workers are set to work on this worker task. If their answers are different from each other, utilize an additional two workers to establish a majority-based answer. Once that majority is reached, pay all of them.
Edit rule
To set rule parameters, follow the steps below:
- Click the Edit Rule link to the right of the Adjudication Rule drop-down box. Depending on the Adjudication Rule selected, there is a specific set of rule variable inputs. You need to check every i-th gold question, Majority value, Worst accuracy level upper limit.
- Select the Adjudication Strategy.
- Set the Rule Variables.
- Click the Save button.
note
The Rule you are saving can be used in other tasks.
You can also edit the rule code by clicking the Switch to Advanced View link. To get detailed info, see the Rules and Rule Templates topic.
You can optionally edit the Price Strategy the same way as the Adjudication Rule.