9.1

Edit

Manual Task lifecycle

WorkFusion platform:

  1. A user creates a task and starts designing and configuring it. Right after the task has been created and configured, the user saves the task. At this point, the task entities are created in the MySQL database on the side of the WorkFusion application, not being posted to the endpoint.

Entries are created in the Run table with the DRAFT status, Question table, and Campaign table.

  1. A user runs a task, the Run status is changed to PROCESSING:

    1. A batch process runs and makes a Web Service API call to the selected endpoint (for example, Amazon Mechanical Turk) to post a HIT for the Crowd workers.
    2. The Adjudication Rule for the HIT determines how many Assignments are posted on mTurk.
    3. For the standard 2 + 1 Rule, two Assignments are posted initially.

  2. Once HIT is posted to the endpoint, the following happens:

    1. Entries are created in the AwsHIt table with processingStatus SUBMITTED and EndpoinTask with the ASSIGNABLE status.

    2. In those tables, each row represents a separate HIT.

    3. The number of HITs is based on the number of records in the input file.

    4. The HitSubmissionDataItem table has actual values from the input file.

  3. A batch process keeps polling mTurk to see if posted assignments are completed.

  4. mTurk only gets back to us when all assignments requested for a particular HIT are completed or disposed.

    -- 1) if the disposal of the hit has been failed, the hit can be re-submitted to the endpoint.
select id into @hitId from AwsHit where hitId = '28929BAA-3C58-4DE4-9E8D-50CC0C21D5EC'; // REPLACE WITH YOURS

update EndpointTask et
set et.status='ERROR', et.awsHitId = NULL
    where et.awsHitId = @hitId;

update AwsHit ah
set ah.submissionDate = NULL, ah.processingStatus = 'READY_TO_SUBMIT', ah.assignments = 1, ah.processesAttemptCount = 0
    where ah.id = @hitId;

update StreamDescription sd
left join Run r on r.uuid = sd.objectUUID
left join AwsHit ah on ah.run_id = r.id
set sd.status='OPEN'
    where ah.id = @hitId;

  5. Once all assignments for a HIT are completed by the Crowd, the processingStatus of the HIT in AwsHIt changes from SUBMITTED to REVIEWABLE. The batch job calls the markHitAsReviewable status updated in the database.

  6. At this point, entries are also created in the AwsHitAssignment and AwsHitAssignmentAnswer tables.

  7. Then the Adjudication Rule fires and tries to determine if the answers can be accepted or if additional assignments need to be created.

    1. If Adjudication is not yet completed (for example, the majority has not been reached or max # of assignments have not been posted), the status in AwsHit remains as REVIEWABLE and additional assignments are sent to mTurk (the HIT is extended).
    2. If Adjudication is not yet completed, even if assignments have been returned, the results are shown in the snapshot or on the View Results tab of the Task Details page.
    3. Look at the code in HitApplicationService.processReviewableHit and RuleExecutor.executeRules.

  8. Once Adjudication is complete for a given record, no further assignments are posted. In the AwsHit table, the hitStatus field is changed to DISPOSED and processingStatus to DONE.

  9. At this point, the user can download the data as an Excel snapshot or view the results on the View Results page.

← PreviousNext →