Edit

Execution queue monitoring

Business Process bot record execution attempt

Configuration:

A reloadable file without restarting the WF platform: ${catalina.home}/conf/reloadable-wf.properties.

The main properties are as follows:

bot.record.execution.track.failed.attempts=true
bot.record.execution.track.released.attempts=false

Behavior

When a bot record execution fails and the track flag is turned on (bot.record.execution.track.failed.attempts=true), it is expected to have record in BotRecordExecutionAttempt.

How it works:

id    hitId   failed  executionStart          executionEnd
1   1       1       2018-08-14 14:20:49.179 2018-08-14 14:20:50.83

When a bot record execution does not call the export plugin and the track flag is turned on (bot.record.execution.track.released.attempts=true), the record is expected to be in BotRecordExecutionAttempt.

id    hitId   failed  executionStart          executionEnd
2   2       0       2018-08-14 15:20:49.179 2018-08-14 15:20:50.838

SQL check for executable bot records:

SELECT * FROM BotRecordExecutionAttempt where hitId=?

Bot record execution queue

SQL check

Executable bot records can be found via the following SQL queries:

Executable bot record

set @global_retry_threshold = 10;
set @now = convert_tz(now(), @@session.time_zone, '+00:00');

SELECT c.title as bp_name, r.rootRunUUID as instance_uuid, r.title as step_title, ah.id as record_id
FROM Run r
JOIN Campaign c on c.id = r.campaign_id
JOIN RunDescription rd ON r.description_id = rd.id 
join AwsHit ah ON ah.run_id = r.id
WHERE r.executingType = 'MACHINE' AND r.status = 'MACHINE_PROCESSING' AND 
      ah.processingStatus = 'SUBMITTED' AND ah.hitOrigination='INTERNAL'
                    AND ((rd.maximumProcessingCount IS NOT NULL AND ah.processesAttemptCount <= rd.maximumProcessingCount)
                          OR (rd.maximumProcessingCount IS NULL AND ah.processesAttemptCount <= @global_retry_threshold)) 
                     AND ah.minimumExecutionDate <= @now;

Executable bot records (plus the Bot Source and record processing info):

SELECT NOW(),r.executingType, r.rootRunUUID, r.title, c.title, mcs.name, mcs.threadCount, mcs.batchSize, mcs.prioritizeEarlierRuns, ah.id, ah.creationTime, ah.expirationDate, ah.processingStatus, ah.prevProcessingStatus, ah.lastProcessingTime, ah.processesAttemptCount, ah.sandboxInd, ah.run_id, ah.completionDate, ah.submissionDate, ah.uuid, ah.sortIndex, ah.minimumExecutionDate, ah.splitted
FROM Run r
join RunDescription rd on r.description_id = rd.id
join Campaign c on r.campaign_id = c.id
join AwsHit ah on ah.run_id = r.id
left join CampaignMap cm on cm.id = r.campaignMap_id
left join Campaign stepCamp on stepCamp.id = cm.campaign
left join MACHINE_CONFIG mc on (mc.id = stepCamp.machineConfigId OR mc.id =
c.machineConfigId)
left join MachineCampaignSource mcs on mcs.id = mc.source_id
WHERE ((ah.processingStatus in ('READY_TO_SUBMIT', 'SUBMITTED') AND
 ((rd.maximumProcessingCount IS NOT NULL
AND ah.processesAttemptCount <= rd.maximumProcessingCount)
 OR (rd.maximumProcessingCount IS NULL
AND ah.processesAttemptCount <= 10))) OR ah.advancedNextStep = NULL)
AND EXISTS (select step.id from Run step where step.rootRunUUID = r.rootRunUUID
and step.status IN ('PROCESSING', 'MACHINE_PROCESSING'))
and r.executingType = 'MACHINE'
order by r.rootRunUUID, cm.stepPosition;

Sample output:

bp_name	Instance_uuid	step_title	record_id
BP empty step	0ddc361d-cb2d-4d44-8a59-f890c4959ae6	Empty step	35414

REST check

Use the following endpoint: api/v1/monitoring/bots/sources

Overview report:

curl -X GET http://localhost:8080/workfusion/api/v1/monitoring/bots/sources --user <user>:<password>

{
    "Member [127.0.0.1]:5702 - 932122e0-ec6e-45d5-aaa2-82ff959c1549 this": [
        {
            "uuid": "anonymous-source-0ddc361d-cb2d-4d44-8a59-f890c4959ae6",
            "threads": 5,
            "runningTaskCount": 1,
            "waitingWriteTaskCount": 0,
            "queuedTaskCount": 0,
            "runningTasks": [],
            "waitingWriteTasks": [],
            "queuedTasks": []
        }
    ]
}

Detailed source report:

curl -X GET http://localhost:8080/workfusion/api/v1/monitoring/bots/sources/anonymous-source-0ddc361d-cb2d-4d44-8a59-f890c4959ae6 --user <user>:<password>

{
    "Member [127.0.0.1]:5702 - 932122e0-ec6e-45d5-aaa2-82ff959c1549 this": [
        {
            "uuid": "anonymous-source-0ddc361d-cb2d-4d44-8a59-f890c4959ae6",
            "threads": 5,
            "runningTaskCount": 1,
            "waitingWriteTaskCount": 0,
            "queuedTaskCount": 0,
            "runningTasks": [
                {
                    "taskUuid": "0ddc361d-cb2d-4d44-8a59-f890c4959ae6",
                    "businessProcessUuid": "0ddc361d-cb2d-4d44-8a59-f890c4959ae6",
                    "hitUuid": "75acc244-6eee-4be6-8775-0021f95afd94",
                    "executionTime": 673452,
                    "queueTime": 673471
                }
            ],
            "waitingWriteTasks": [],
            "queuedTasks": []
        }
    ]
}

BP lifecycle task queue

Tasks are processed via a thread pool of a configurable size (3 by default). Only one task from a BP instance is processed simultaneously. For a large number of simultaneous BP instances, it can be increased.

workfusion.properties

#how many parallel threads to use for BP state transitions, e.g. human HIT submission, step completion, step transition, etc
bp.step.transition.pool.size=3

A task produces the following:

• Step stats update:

  select c.title as process_name, r.rootRunUUID as instance_uuid, r.title as step_name, r.id as step_id
  from Run r
  join Campaign c on c.id = r.campaign_id
  where r.`status` in ('MACHINE_PROCESSING', 'PROCESSING') and r.submissionSize > r.competedSubmissions
  order by r.rootRunUUID;
  

  process_name	instance_uuid	step_name	step_id
  Task with workforce data	027a0804-258f-485b-86ee-47148f67e2d9	Task with workforce data	452
  Automation process: CZPffQnpbn	0579c068-bc1d-41c3-8cb1-507e052b5aa5	(Generic) Upload training set	4
  BP empty step	0ddc361d-cb2d-4d44-8a59-f890c4959ae6	Empty step	600

• Manual step record transition:

  set @global_retry_threshold = 20;
  SELECT DISTINCT e.runUUID FROM AwsHitProcessingEvent e WHERE e.processingCount < @global_retry_threshold;
  

• Submission of Manual Tasks to a Crowd

  SELECT root.uuid as instance_uuid, r.title as step_name, r.uuid as step_uuid
  FROM StreamDescription sd 
  JOIN Run r ON r.uuid = sd.objectUUID 
  JOIN Run root on r.rootRunUUID = root.uuid 
  WHERE sd.status = 'OPEN' AND r.status IN ('READY_TO_PROCESS', 'PROCESSING') 
        AND (root.businessProcessStatus IS NULL OR root.businessProcessStatus = 'PROCESSING')
  ORDER BY sd.priority DESC
  

  instance_uuid	step_name	step_uuid
  8f94b6aa-579d-471f-a0ca-23b3b442c7a7	[Regression] Gold accuracy	8f94b6aa-579d-471f-a0ca-23b3b442c7a7

• Bot step transition or completion

  SELECT c.title as bp_name, r.rootRunUUID as instance_uuid, r.title as step_name, r.id AS step_id
  FROM Run r
  JOIN Campaign c on c.id = r.campaign_id
  WHERE r.submissionSize > 0 AND r. STATUS = 'MACHINE_PROCESSING' AND r.executingType = 'MACHINE'
  AND (
      EXISTS (
          SELECT hit.id
          FROM AwsHit hit
          WHERE hit.run_id = r.id
          AND hit.processingStatus = 'DONE'
          AND hit.advancedNextStep IS NULL
      ) OR (r.competedSubmissions = r.submissionSize)
  )
  

  bp_name	instance_uuid	step_name	step_id
  Split-join bot	000c92d8-00cc-49c4-add1-fa3cd5d7855d	Final Results	17592
  Split-join bot	d6054a53-1331-426e-bf40-fe8586729f5f	Final Results	19504

Performance metrics

Available in Useful logs and how to check them.