Edit

Configure ETL Bot

ETL Bot Feature

In some cases from time to time you should make minor adjustments in configurations of Business Process steps. You may need such easy-configurable options for every launch or every week or month.

It's not a problem for a developer to open a Bot and change a few variable values there. But it's another story when an in-production Business Process is managed by Operations SME or similar non-technical persons. Such people do not dive too deep in Bot structure, can get scared to modify the code or can modify it in a wrong way.

To avoid such inconveniences WorkFusion provides the ETL Bot feature. Once a Bot s converted to an ETL Bot, the end users of Business Process get an editable form with HTML-like fields instead of Groovy/XML code. So, an easy and error-proof configurability is achieved.

Rather than enabling simplicity of configuration, an ETL Bot can encapsulate a portion of variables transformation and/or additional calculations. It means, a user provides the required data and then they are converted into required variables for <export>. So, an ETL Bot consists of two parts:

• UI form with easy-to-configure parameters.
• Additional Groovy or XML code you require to prepare variables for the next steps.

Perform setup

You can start either with an existing Bot or a new one. Let's consider a simple example: you need to add ETL Bot with three editable fields:

1. If a Bot is already created, copy the Bot source code to the clipboard.

2. Go to Campaigns > Templates and Create a new template. Enter Template Name. For Template Type select Bot Base. Paste your existing Bot code from the clipboard. If the template is new as in our example, use the following code:

   <?xml version="1.0" encoding="UTF-8"?>
   <config xmlns="http://web-harvest.sourceforge.net/schema/1.0/config" scriptlang="groovy">
   
       <var-def name="var1">
           {{etl_column_name_var1}}
       </var-def>
     
       <var-def name="var2">
           {{etl_column_name_var2}}
       </var-def>
     
       <var-def name="var3">
           {{etl_column_name_var3}}
       </var-def>
        
           
       <script><![CDATA[
           // Your custom conversion code goes here
       ]]></script>
     
       <insert-datastore datastore-name="${datastoreName}" create="true">
           <script return="filteredBotVars"/>
       </insert-datastore>
   
      <export include-original-data="true">
          <single-column name="var1" value="{{etl_column_name_var1}}"></single-column>
          <single-column name="var2" value="{{etl_column_name_var2}}"></single-column>
          <single-column name="var3" value="{{etl_column_name_var3}}"></single-column>
      </export>
   
   </config>
   

   Make sure you have each UI field, for example, {{etl_column_name_var3}} defined in the <export> section. Otherwise, such field will not appear on the UI form of the ETL Bot

3. Go to Configuration > Use Cases. Select the Bot tab and then Create Use Case. Enter Name, select Category where your Use Case will appear for reuse. In Base Template, select a template created in step 2. In Type, select ETL.

4. Configure all variables you want to have as fields in the UI. To do so, use the right panel. Click Add Answer to add field by field. In this example, you are adding three Free Text variables:

5. Once Use Case is created, you can use it in any Business Process. Open the Business Process from the Business Processes list to edit or create a new one. In the Design tab, drag a new Bot into canvas and double-click the Bot square bar. Control Tower will ask you to choose a Use Case to start from. Use the recently created Use Case from step 4.

6. An ETL Bot has two modes: BASIC EDITOR (opens by default) and ADVANCED EDITOR. Make sure you see three input fields for our example in the BASIC mode:

ETL Bot examples

Now, modify your initial example and add a piece of custom logic to the ETL Bot you created.

Let's assume that our task is to ask a user to provide variable names and save those variable values to Control Tower's Data Store. Once done, you need to output the variables to the next Business Process step. The modified ETL Bot is as below:

<?xml version="1.0" encoding="UTF-8"?>
<config xmlns="http://web-harvest.sourceforge.net/schema/1.0/config" scriptlang="groovy">

    <var-def name="var1">
        {{etl_column_name_var1}}
    </var-def>
  
    <var-def name="var2">
        {{etl_column_name_var2}}
    </var-def>
  
    <var-def name="var3">
        {{etl_column_name_var3}}
    </var-def>
     
    <!-- DO NOT change code below //-->
    <!-- List of 3 DataStore default column names //-->
    <var-def name="datastoreColumns">var1|var2|var3</var-def>    
    <!-- DataStore name where data will land //-->
    <var-def name="datastoreName">CustomDashboardData</var-def>
    <!-- DataStore column name for Run's unique ID //-->
    <var-def name="columnUUIDName">uuid</var-def>
    <!-- DataStore column name for BP's unique Name //-->
    <var-def name="columnBPName">bp_name</var-def>
        
    <script><![CDATA[
        includedVars = [var1.getWrappedObject().get(0).toString(), var2.getWrappedObject().get(0).toString(), var3.getWrappedObject().get(0).toString()];
        dsColumns = datastoreColumns.toString().tokenize('|');
        allBotVars = hit_submission_data_item.getWrappedObject().getItemValueMap();
        filteredBotVars = allBotVars.subMap(includedVars);
        (0..2).each { filteredBotVars.put(dsColumns[it], filteredBotVars.remove(includedVars[it])); }
        filteredBotVars.put(columnBPName.toString(), item.getWrappedObject().getRun().getCampaignName());
        filteredBotVars.put(columnUUIDName.toString(), item.getWrappedObject().getRun().getRootRunUuid());
    ]]></script>
  
    <insert-datastore datastore-name="${datastoreName}" create="true">
        <script return="filteredBotVars"/>
    </insert-datastore>

   <export include-original-data="true">
       <single-column name="var1" value="{{etl_column_name_var1}}"></single-column>
       <single-column name="var2" value="{{etl_column_name_var2}}"></single-column>
       <single-column name="var3" value="{{etl_column_name_var3}}"></single-column>
   </export>

</config>

As a result, for a Business Process user, the ETL Bot in the BASIC mode looks exactly as shown on the screenshot from step 6 above.

In addition, in run-time, this Bot applies the user configuration to the custom code you added, for example, the variables typed in by the user are stored in a Data Store and output into <export>.