10.1.2

Edit

Set up data flow in Business Process

This article describes how to pass data between Business Process (BP) steps, both manual and bot ones.

The key concept of Control Tower BPs is to get data to flow between steps while processing big amounts of uniform data. Input data for Business Processes is divided into items with a defined structure that are called records. These records can be listed in a CSV file or dynamically grabbed from a Data Store, search result, parsed from a list, and so on.

Each step can do the following:

  • Add new input data columns
  • Include or exclude data from previous steps
  • Split or merge data (increase or decrease the number of records)

Manual Tasks and composite rule

The simple BP is intended to detect document language and extract special fields for all English invoices.

You can download these files, import, and run the BP in your Control Tower:

Process details

Input file

The input file contains 6 records (documents) with 2 columns:

  • document_image: links to the PNG scans of invoices
  • document_text: links to the HTML code of invoices

These links will be used in Manual Tasks as input data. They are accessible in Manual Tasks using the ${question.data['column_name']} syntax:

  • ${question.data['**document_image**']}
  • ${question.data['**document_text**']}

Manual step 1: Detect Language

The first step uses the image address and link address column data as input.

The Unique codes for the two answers (language, proforma) will serve as output data provided by workers on the task submission.

The Manual Task code sample below shows how to use variables in text and in HTML attributes (for images and links):

Manual Task code

<!-- Using variables in image and link attributes -->


<@editable id="__DATA__">
    <img src="${question.data['document_image']}" width="500"><br><br>
    <p>Document Text - <a href="${question.data['document_text']}" >${question.data['document_text']}</a></p>
</@editable>

Composite rule

The composite rule serves to route all English invoices to the second manual step by comparing the answer from the first step (language) to the English string.

Alternatively, the proforma answer could be used in the rule condition.

Manual step 2: Extract Data

This step uses the following columns as input data:

  • language (from the 1st step)
  • proforma (from the 1st step)
  • document_text (from the CSV input file) as source text for the Information Extraction answer.

To use a data column as source text for the Information Extraction answer, enter its name in the Unique Code field:

On the Task Preview page, you cannot view the data from the first step (language, proforma) until at least one record is submitted on the first step.

After you run the BP and submit several worker tasks from step 1, this data is populated on the Task Preview page. The tasks from Step 2 are available for accepting in WorkSpace:

Results

You can view data for each step in the View Results > Data page. The Final Results option contains columns at the process end (after the last step is completed):

  • Input data columns:
    • document_text
    • document_image
  • Step 1 (Detect Language) output:
    • proforma
    • language
  • Step 2 (Extract Data) output:
    • document_tagged_text contains the IE text with tags created by the worker
    • number
    • date
    • items
      • name
      • price

Note that Control Tower adds calculated _confidence and _accuracy columns for each Manual Task answer, which can be disabled by scrolling the table to the right and clicking the cogwheel icon.

Bot Tasks and Export plugin

The following process is intended to collect and validate official websites for a given list of companies. Valid websites are saved to a Data Store, invalid websites are emailed to a defined email address.

You can download these files, import, and run the BP in your Control Tower:

Process details

Input file

The input file contains five records (company profiles) with a single column:

  • company_name stands for the name of the company to search the official website for.

These links will be used in Manual Tasks as input data.

Manual step 1: Find Website by Company Name

The first Manual step is designed to show the company name and link to a default Google search for this company. A worker is expected to analyze the Google search, pick up the right website URL from it, and submit this URL.

Note that the same input data column is used 3 times here:

  • As text to display a company name:

    ${question.data['company_name']}

  • As a part of link href attribute and link text:

    <a href="https://www.google.com/search?q=website+${question.data['company_name']}" target="_blank">https://www.google.com/search?q=website+${question.data['company_name']}</a>

The task output is provided by the answer with the website_url unique code. The Task Preview shows the following:

This Manual Task also has a hidden answer with the worker_name unique code that captures the worker's first and last names.

Code snippet to get worker details

<#if run??>
        <input type="hidden" name="${question.identifier!}_worker_name_skipCheck"/>

        <script type="text/javascript">
            $(document).ready(function() {
              $("input[name=${question.identifier!}_worker_name_skipCheck]").val("${worker.firstName}" + " ${worker.lastName}");
            });
        </script>

</#if>

See more information in Template Variables.

Bot step 2: Check URL

The second process step is a Bot Task intended to accomplish the following:

  1. Make a GET HTTP request to the website_url submitted by the worker on the previous step. The variable value is accessed by using the ${website_url} syntax.
  2. Check the response code using the http.statusCode object property.
  3. If the response code is 200, set the validity variable to true, otherwise to false.
  4. Export the validity variable value to a new column with the same name. This column value will be used in the next composite rule to decide where to route the record: to an Exception Handling step or to Data Store logging step.

Checking website response using http plugin

<?xml version="1.0" encoding="UTF-8"?>
<config xmlns="http://web-harvest.sourceforge.net/schema/1.0/config" scriptlang="groovy">

    <required name="website_url"/>

    <var-def name="validity"/>
    <try>
        <body>
            <http url="${website_url}"/>
            <var-def name="validity">
                <template>${http.statusCode == 200}</template>
            </var-def>
        </body>
        <catch>
            <var-def name="validity">
                false
            </var-def>
        </catch>
    </try>

    <export include-original-data="true">
        <single-column name="validity" value="${validity}"/>
    </export>

</config>

Composite rule

This rule checks the validity column and routes records to the Exception Handling step or to Data Store logging step.

Bot step 3: Write Results to Data Store

This step takes all data columns from the input file and previous steps and writes their values to the websites Data Store using Data Store plugins. If this Data Store does not exist, it will be created automatically.

Note that the column values are accessible in the <script> block by their names: website\_url.toString().

Writing results to Data Store

<?xml version="1.0" encoding="UTF-8"?>
<config xmlns="http://web-harvest.sourceforge.net/schema/1.0/config" scriptlang="groovy">

    <script><![CDATA[
        insertedParamMap = new HashMap();
        insertedParamMap.put("company_name", company_name.toString());
        insertedParamMap.put("website_url", website_url.toString());
        insertedParamMap.put("validity", validity.toString());
        jsonValueMap = new com.google.gson.Gson().toJson(insertedParamMap);
    ]]></script>

    <insert-datastore datastore-name="websites" json-value-map="\${jsonValueMap}" create="true"/>

    <export include-original-data="true"></export>
</config>

The step results look like this:

Note that not all the records are saved to the Data Store because the rest of the records had invalid website_url and were routed to the Error Handling step.

Bot step 4: Error Handling - Send Email

This step is intended to send an email to a specific address (recepient@email.com in the example) containing the following info:

  • Company name: ${company_name}
  • Name of the worker who submitted the invalid URL: ${worker_name}
  • Invalid website URL: ${website_url}
  • Link to a particular Business Process instance for quick navigation: ${bp_link} variable. See more details in Apply WebHarvest and WorkFusion context variables.

Sending email in a Bot Task

<?xml version="1.0" encoding="UTF-8"?>
<config xmlns="http://web-harvest.sourceforge.net/schema/1.0/config" scriptlang="groovy">

  <var-def name="bp_link">
   <template>${applicationHost}/workfusion/secure/business-process/edit/${item.getWrappedObject().getRun().getRootRunUuid()}</template>
  </var-def>

  <secure-store-get alias="my-keys"/>

  <script><![CDATA[
        Map entryMap = secureEntryMap.getWrappedObject();
        com.freedomoss.crowdcontrol.webharvest.web.dto.SecureEntryDTO obj = entryMap.get("my-keys");
        username = obj.getKey().toString();
        password = obj.getValue().toString();
   ]]></script>

    <mail smtp-host="smtp.your-smtp-server.com"
        smtp-port="465"
        type="text"
        from="${username}"
        to="recepient@email.com"
        subject="Website not found"
        charset="UTF-8"
        username="${username}"
        password="${password}"
        security="ssl">

        <template>Website not found for: ${company_name}</template>
        <template>User ${worker_name} submitted this invalid URL: ${website_url}</template>
        <template>Business Process link: ${bp_link}</template>

    </mail>

    <export include-original-data="false">
    </export>

</config>

The following example shows real emails sent by Control Tower with variables substituted by real data:

The ${username} and ${password} variable values are taken from the Secrets Vault using the Secrets Vault Plugins. This is a safe way not to expose sensitive data in the Bot Task code.

You need to create an entry in Secrets Vault with the my-keys name.

The current step contains variables with sensitive data, therefore it is better not to pass its variables to the export plugin.

Note that the export plugin has the include-original-data="false" attribute. It means that all information from previous steps will be deleted (will not be in final process results).

Execution and results

When you run the Business Process and submit all Manual Tasks, all bot steps will be executed automatically. As you can see on the screenshot below, two records had invalid URLs and were routed to the Error Handling step. The other three records were logged to the Data Store.

Results of a manual step:

The Check URL bot step validated the URLs and added the check result to the validity column:

As you can see from the Final Results, no data were saved for the two records that went to the Error Handling step. Their cells are just blank because of the include-original-data="false" attribute and empty export section.

← View Business Process and data flowsNext →