next

Edit

Set up data flow in Business Process

This article describes how to pass data between Business Process (BP) steps, both manual and bot ones.

The key concept of Control Tower BPs is to get data to flow between steps while processing big amounts of uniform data. Input data for Business Processes is divided into items with a defined structure that are called records. These records can be listed in a CSV file or dynamically grabbed from a Data Store, search result, parsed from a list, and so on.

Each step can do the following:

  • Add new columns
  • Include or exclude data from previous steps
  • Split or merge data (increase or decrease the number of records)

Manual Tasks and composite rule

The simple BP is intended to detect the document language and extract special fields for all English invoices.

You can download these files, import, and run the BP in your Control Tower instance:

Process details

Input file

The input file contains six records (documents) with two columns:

  • document_image: links to the PNG scans of invoices
  • document_text: links to the HTML code of invoices

These links are used in Manual Tasks as input data. They are accessible in Manual Tasks using the ${question.data['column_name']} syntax:

  • ${question.data['**document_image**']}
  • ${question.data['**document_text**']}

Manual Step 1: detect language

The first step uses the data from the following two columns as input:

  • Image address
  • Link address

The Unique codes for the two answers (language, proforma) will serve as output data provided by workers on the task submission.

The code sample shows how to use variables in text and in HTML attributes for images and links:

<!-- Using variables in image and link attributes -->


<@editable id="__DATA__">
    <img src="${question.data['document_image']}" width="500"><br><br>
    <p>Document Text - <a href="${question.data['document_text']}" >${question.data['document_text']}</a></p>
</@editable>

Composite rule

The composite rule serves to route all English invoices to the second manual step by comparing the answer from the first step (language) to the English string.

Alternatively, the proforma answer could be used in the rule condition.

Manual Step 2: extract data

This step uses the following columns as input data:

  • language from the first Step
  • proforma from the first Step
  • document_text from the input CSV file as the source text for the Information Extraction answer

To use a data column as the source text for the Information Extraction answer, enter its name in the Unique Code field:

On the Task Preview page, you cannot view the data from the first step (language, proforma) until at least one record is submitted on the first step.

After you run the BP and submit several worker tasks from Step 1, you can see the data on the Task Preview page. The tasks from Step 2 are available for accepting in WorkSpace:

Results

You can view data for each step in the View Results > Data page. The Final Results option contains columns at the process end (after the last step is completed):

  • Input data columns:
    • document_text
    • document_image
  • Step 1 (Detect Language) output:
    • proforma
    • language
  • Step 2 (Extract Data) output:
    • document_tagged_text contains the IE text with tags created by the worker
    • number
    • date
    • items
      • name
      • price

Bot Tasks and Export plugin

The following process is intended to collect and validate official websites for a given list of companies. Valid websites are saved to a Data Store, and invalid websites are emailed to a defined email address.

You can download these files, import, and run the BP in your Control Tower:

Process details

Input file

The input file contains five records (company profiles) with a single column:

  • company_name: names of the companies for which you need to find official websites.

These links are used in Manual Tasks as input data.

Manual step 1: find website by company name

The first Manual step is designed to show a company name and the link to a default Google search for the company. A worker is expected to analyze the Google search, pick up the right website URL from it, and submit the link.

Note that the same input data column is used three times here:

  • As text to display a company name:

    ${question.data['company_name']}

  • As a part of link href attribute and link text:

    <a href="https://www.google.com/search?q=website+${question.data['company_name']}" target="_blank">https://www.google.com/search?q=website+${question.data['company_name']}</a>

The task output is provided by the answer with the website_url unique code. The Task Preview page shows the following:

This Manual Task also has a hidden answer with worker_name unique code that captures the worker's first and last names.

Code snippet to get worker details

<#if run??>
        <input type="hidden" name="${question.identifier!}_worker_name_skipCheck"/>

        <script type="text/javascript">
            $(document).ready(function() {
              $("input[name=${question.identifier!}_worker_name_skipCheck]").val("${worker.firstName}" + " ${worker.lastName}");
            });
        </script>

</#if>

See more information in Template variables.

Bot Step 2: check URL

The second process step is a Bot Task intended to do the following:

  1. Make a GET HTTP request to the website_url submitted by the worker on the previous step. The variable value is accessed by using the ${website\_url} syntax.
  2. Check the response code using the http.statusCode object property.
  3. If the response code is 200, set the validity variable to true, otherwise to false.
  4. Export the validity variable value to a new column with the same name. This column value will be used in the next composite rule to decide where to route the record: to an Exception Handling step or to Data Store logging step.

Checking website response using http plugin

<?xml version="1.0" encoding="UTF-8"?>
<config xmlns="http://web-harvest.sourceforge.net/schema/1.0/config" scriptlang="groovy">

    <required name="website_url"/>

    <var-def name="validity"/>
    <try>
        <body>
            <http url="${website_url}"/>
            <var-def name="validity">
                <template>${http.statusCode == 200}</template>
            </var-def>
        </body>
        <catch>
            <var-def name="validity">
                false
            </var-def>
        </catch>
    </try>

    <export include-original-data="true">
        <single-column name="validity" value="${validity}"/>
    </export>

</config>

Composite rule

This rule checks the validity column and routes records to the Exception Handling step or to Data Store logging step.

Bot Step 3: write results to Data Store

The step takes all data columns from the input file and previous steps and writes their values into the websites Data Store using Data Store plugins. If the Data Store does not exist, it is created automatically.

Note that you can get the column values in the <script> block by their names: website_url.toString().

Writing results to Data Store

<?xml version="1.0" encoding="UTF-8"?>
<config xmlns="http://web-harvest.sourceforge.net/schema/1.0/config" scriptlang="groovy">

    <script><![CDATA[
        insertedParamMap = new HashMap();
        insertedParamMap.put("company_name", company_name.toString());
        insertedParamMap.put("website_url", website_url.toString());
        insertedParamMap.put("validity", validity.toString());
        jsonValueMap = new com.google.gson.Gson().toJson(insertedParamMap);
    ]]></script>

    <insert-datastore datastore-name="websites" json-value-map="\${jsonValueMap}" create="true"/>

    <export include-original-data="true"></export>
</config>

The step results look as follows:

Note that not all records are saved to the Data Store because some had invalid website_url and were routed to the Error Handling step.

Bot Step 4: error handling and sending emails

This step is intended to send an email to a specific address (recepient@email.com in the example) containing the following info:

  • Company name: ${company_name}.
  • Name of the worker who submitted an invalid URL: ${worker_name}.
  • Invalid website URL: ${website_url}.
  • Link to a particular Business Process instance for quick navigation: ${bp_link} variable. For details, see Apply WebHarvest and WorkFusion context variables.

Sending email in a Bot Task

<?xml version="1.0" encoding="UTF-8"?>
<config xmlns="http://web-harvest.sourceforge.net/schema/1.0/config" scriptlang="groovy">

  <var-def name="bp_link">
   <template>${applicationHost}/workfusion/secure/business-process/edit/${item.getWrappedObject().getRun().getRootRunUuid()}</template>
  </var-def>

  <secure-store-get alias="my-keys"/>

  <script><![CDATA[
        Map entryMap = secureEntryMap.getWrappedObject();
        com.freedomoss.crowdcontrol.webharvest.web.dto.SecureEntryDTO obj = entryMap.get("my-keys");
        username = obj.getKey().toString();
        password = obj.getValue().toString();
   ]]></script>

    <mail smtp-host="smtp.your-smtp-server.com"
        smtp-port="465"
        type="text"
        from="${username}"
        to="recepient@email.com"
        subject="Website not found"
        charset="UTF-8"
        username="${username}"
        password="${password}"
        security="ssl">

        <template>Website not found for: ${company_name}</template>
        <template>User ${worker_name} submitted this invalid URL: ${website_url}</template>
        <template>Business Process link: ${bp_link}</template>

    </mail>

    <export include-original-data="false">
    </export>

</config>

The following example shows real emails sent by Control Tower with variables substituted by real data:

The ${username} and ${password} variable values are taken from the Secrets Vault using the Secrets Vault Plugins. This is a safe way not to expose sensitive data in the Bot Task code.

You need to create an entry in Secrets Vault with the my-keys name.

The current step contains variables with sensitive data. Therefore, it is better not to pass its variables to the export plugin.

Note that the export plugin has the include-original-data="false" attribute. It means that all information from previous steps is deleted (will not be present in the final process results).

Execution and results

When you run the Business Process and submit all Manual Tasks, all bot steps are executed automatically. As you can see on the screenshot below, two records had invalid URLs and were routed to the Error Handling step. Other three records were logged to the Data Store.

Manual step results:

The Check URL bot step validated the URLs and added the check result to the validity column:

As you can see in Final Results, no data were saved for two records that went to the Error Handling step. Their cells are just blank because of the include-original-data="false" attribute and empty export section.

← View Business Process and data flowsProcess transactions with REST connector →