Edit

Configure ETL Bot

View ETL Bot feature goals

In some cases, you should make minor adjustments in configurations of Business Process steps from time to time. You may need such easy-configurable options for each launch or every week or month.

A developer doesn't face a problem opening a Bot and changing a few variable values there. But it's another story when an in-production Business Process is managed by Operations SMEs or similar non-technical personas. Such people don't dive too deep into the Bot structure, get scared to modify the code, or can modify it in the wrong way.

To avoid such inconveniences, WorkFusion provides the ETL Bot feature. Once a Bot gets converted to an ETL Bot, the end-users of Business Process get an editable form with HTML-like fields instead of Groovy or XML code. So, you achieve easy and error-proof configurability.

Rather than enabling configuration simplicity, an ETL Bot can encapsulate a portion of variables transformation and additional calculations. It means a user provides the required data, which gets converted into required variables for <export>. So, an ETL Bot consists of two parts:

• UI form with easy-to-configure parameters
• Additional Groovy or XML code you require to prepare variables for the next steps

Set up ETL Bot

You can start either with an existing Bot or a new one. Consider a simple example and set up an ETL Bot with three editable fields:

1. If a Bot is already created, copy the Bot source code to the clipboard.

2. Go to Advanced > Templates.

3. Click Create to add a new template.

   

   1. Enter Template Name. For Template Type, select Bot Base.

   

   2. Paste your existing Bot code from the clipboard. If the template is new as the following example, use the following code:

   <?xml version="1.0" encoding="UTF-8"?>
       <config xmlns="http://web-harvest.sourceforge.net/schema/1.0/config" scriptlang="groovy">
   
           <var-def name="var1">
           {{etl_column_name_var1}}
           </var-def>
         
           <var-def name="var2">
           {{etl_column_name_var2}}
           </var-def>
         
           <var-def name="var3">
           {{etl_column_name_var3}}
           </var-def>
            
           <script><![CDATA[
               // Your custom conversion code goes here
           ]]></script>
         
           <insert-datastore datastore-name="${datastoreName}" create="true">
               <script return="filteredBotVars"/>
           </insert-datastore>
   
           <export include-original-data="true">
               <single-column name="var1" value="{{etl_column_name_var1}}"></single-column>
               <single-column name="var2" value="{{etl_column_name_var2}}"></single-column>
               <single-column name="var3" value="{{etl_column_name_var3}}"></single-column>
           </export>
       </config>
   

   3. Click Save.

   

   caution

   Make sure that in the <export> section, you have each UI field defined, for example, {{etl_column_name_var3}}. Otherwise, the field doesn't appear on the UI form of the ETL Bot.

4. Go to System Settings > Operations.

5. Select the Bot tab and click Create Operation.

   

   1. Enter Name.
   2. Click Show Advanced Options to continue with the Operation settings.

   

   • In Category, select the category where your Operation appears for reuse.
   • In Base template, select the template created in step 3.
   • In Type, select ETL.
   3. At this point, you need to configure all the variables you want to have as fields in the UI. To do so, click Add Answer and add the data field by field. Click Save Answer when finished. In this example, you add three Free Text variables:

   

   4. Click Save.

6. Once the Operation gets created, you can use it in any Business Process. On the Business Processes tab, click View All to view the list of existing Business Processes. Click Create to create a new Business Process.

   

7. Within the Business Process, go to the Workflow tab and click Edit Process. Add a new Bot to the workflow and then double-click the newly added Bot square bar.

8. On the Operations tab, Control Tower displays Operation categories. Select a category and within the category, select the recently created Operation.

   

   

9. On the Design tab, add the name and configuration details of the new Bot. Click Save.

   note

   The ETL Bot has two modes: default BASIC EDITOR and ADVANCED EDITOR. Ensure you see three input fields for the example in the BASIC mode.

   

Check examples

Now, you can modify the initial example and add a piece of custom logic to the ETL Bot you have created.

Let's assume that your task is to ask a user to provide variable names and save those variable values to a Control Tower's Data Store. Once done, you need to output the variables to the following Business Process step.

Expand to view the modified ETL Bot

<?xml version="1.0" encoding="UTF-8"?>
<config xmlns="http://web-harvest.sourceforge.net/schema/1.0/config" scriptlang="groovy">

    <var-def name="var1">
        {{etl_column_name_var1}}
    </var-def>
  
    <var-def name="var2">
        {{etl_column_name_var2}}
    </var-def>
  
    <var-def name="var3">
        {{etl_column_name_var3}}
    </var-def>
     
    <!-- DO NOT change code below //-->
    <!-- List of 3 DataStore default column names //-->
    <var-def name="datastoreColumns">var1|var2|var3</var-def>    
    <!-- DataStore name where data will land //-->
    <var-def name="datastoreName">CustomDashboardData</var-def>
    <!-- DataStore column name for Run's unique ID //-->
    <var-def name="columnUUIDName">uuid</var-def>
    <!-- DataStore column name for BP's unique Name //-->
    <var-def name="columnBPName">bp_name</var-def>
        
    <script><![CDATA[
        includedVars = [var1.getWrappedObject().get(0).toString(), var2.getWrappedObject().get(0).toString(), var3.getWrappedObject().get(0).toString()];
        dsColumns = datastoreColumns.toString().tokenize('|');
        allBotVars = hit_submission_data_item.getWrappedObject().getItemValueMap();
        filteredBotVars = allBotVars.subMap(includedVars);
        (0..2).each { filteredBotVars.put(dsColumns[it], filteredBotVars.remove(includedVars[it])); }
        filteredBotVars.put(columnBPName.toString(), item.getWrappedObject().getRun().getCampaignName());
        filteredBotVars.put(columnUUIDName.toString(), item.getWrappedObject().getRun().getRootRunUuid());
    ]]></script>
  
    <insert-datastore datastore-name="${datastoreName}" create="true">
        <script return="filteredBotVars"/>
    </insert-datastore>

   <export include-original-data="true">
       <single-column name="var1" value="{{etl_column_name_var1}}"></single-column>
       <single-column name="var2" value="{{etl_column_name_var2}}"></single-column>
       <single-column name="var3" value="{{etl_column_name_var3}}"></single-column>
   </export>

</config>

As a result, for a Business Process user, the ETL Bot in the BASIC mode looks identical to the screenshot from step 9. In addition, this Bot applies the user configuration to the custom code you have added in run-time, meaning the variables typed in are getting stored in the Data Store and the output into export.