Apply Bot Task plugins

By default, you can use bot plugins included in the original WebHarvest framework. To learn more, refer to Standard WebHarvest processors. To create WebHarvest XML configs, you can also use the plugins implemented additionally in the Java language by the WorkFusion team.

XML configuration is executed for each submission. If the input CSV file contains 40 records, the task consists of 40 submissions, respectively. The WebHarvest XML configuration is executed 40 times—once for each submission.

WorkFusion puts data from the CSV file into the WebHarvest context, so you can access it using placeholders.

There is a set of defined variables for you to use and access in a WebHarvest XML configuration. For more information, see Bot Task context.

Plugin	Description
cache	Obtains an object from the global server cache by a key or stores an object in the cache for future requests.
conversion	Converts input data into a custom format.
datastore	Manages Data Stores and transactions.
date-format	Brings the string representation of a date to the desired format.
to-from	Reads data from XLSX or XLS formats and writes to List<Map<String, Object>> [row:[columnName:value]] and vice versa.
excel-to-csv	Converts an Excel file into a CSV one.
export	Stores information collected during Bot Task execution in the output file. Requires at least one child element.
http-extended	Backports the http plugin from the WebHarvest trunk codebase.
include-config	Executes a specified WorkFusion bot configuration within the scope of the current execution (including recursive executions).
JSON manipulation	Converts a JSON string into an object, searches in JSON, deletes and adds nodes, changes node values.
language-extractor	Detects the language of a given website.
list-to-csv	Exports JSON-formatted info to a given CSV file.
log	Logs messages you can see in Events of a Run and log files.
mail-check	Connects to a mail server and checks for new unread emails with a specified subject pattern.
OCR	Recognizes text in images using OCR Service and ABBYY FREngine.
pool	Semaphore functionality for objects in the list section. For each invocation, the plugin returns any unborrowed object from the evaluated list.
release	Postpones the subsequent execution of the current record.
required	Declares which column names (variables) are to be passed to the input data file.
robotics	Clicks through desktop or web applications using the commands in WebHarvest scripts.
S3	Accesses and manages data on the Amazon S3 storage.
Secrets Vault	Provides the capabilities to manage Secrets Vault.
script-var	An alternative to the var-def plugin in scripts to simplify access to the resulting value.
similarity-score	Compares two text strings and provides their similarity score. As an output, the plugin returns a double number, for example, 0.27 or 40.0.
task-start	Starts a task or a Business Process from a specific definition with new input data.
to-text	Parses the content and extracts text from it using the Apache Tika toolkit.
unzip	Extracts the content of zip files.
url-validator	A generic plugin for data validation.
validate	Detects availability of a given URL.
var-global	Inserts the values of the variables defined in the Global Variables Data Store.

---

cache

Open plugin description

The cache plugin is intended to obtain an object from the global server cache by a key or to store an object in the cache for future requests. The plugin body is executed only when the key cannot be found in the cache. Otherwise, the last executed result is returned.

• The current implementation uses a single cache shared inside JVM (the WorkFusion application).
• The cache is shared between all users, but its element lifetime is limited to 10 mins by default.
• The maximum cache size is 50 elements, a new cache key deletes the oldest one.

The plugin contains the following attributes:

Name	Required	Description
key	Yes	Key in the cache. If this key exists, its value is returned. If this key is not found, the script body is executed.
return	No	Variable name where the result is recorded.
language	No	Language for ScriptEngine, for example, Groovy.

Example: You can store the login token in the cache to prevent errors when multiple Bot configs try to log in to a system under the same credentials through REST API.

Auth token example:

<var-def name="auth_token">
    <cache key="Your.Custom.Key" return="a_token">
    <!--  plugin body will be executed if the key is not found in global cache  -->
        <!-- do login -->
        <var-def name="login_response">
            <json-to-xml>
                <http url="${api_url}/login" method="POST">
                    <http-param name="user"><template>${operator_login}</template></http-param>
                    <http-param name="password"><template>${operator_password}</template></http-param>
                </http>
            </json-to-xml>
        </var-def>
        <!-- get authentication tocken and store for future requests -->
        <var-def name="a_token">
            <xpath expression="//data/authToken/text()">
                <template><![CDATA[<root>${login_response}</root>]]></template>
            </xpath>
        </var-def>
    </cache>
</var-def>

Alternative example: You can store some temporary needed information using the cache plugin instead of saving it to the snapshot using the export plugin if a six-minute lifetime fits your Business Process, for example, several Bot steps.

Caching an HTTP request result: There is no need to create multiple HTTP requests to obtain the same information.

Request example:

<?xml version="1.0" encoding="UTF-8"?>
<config xmlns="http://web-harvest.sourceforge.net/schema/1.0/config" scriptlang="groovy">
    
    
    <var-def name="response">
        <cache key="Your.Custom.Key" return="nasa_news">

            <script return="now"><![CDATA[
                now = new Date().toString();
            ]]></script>

            <var-def name="nasa_news">
                <http url="https://api.nasa.gov/planetary/apod?api_key=5HWyNO66SqlGPWoCGYZthgcgvA6XFoJNKhpOKHQC"></http>
                <template>${now}</template>
            </var-def>

        </cache>
    </var-def>
    
    <export include-original-data="true">
        <single-column name="response" value="${response}"/>
    </export>

</config>

When you run a Business Process with multiple records, all these records have the same response value with the same time and JSON (taken from the first record execution).

Response example:

<cache key="SZ.Send.Email.Once.${item.getWrappedObject().getRun().getRootRunUuid()}" return="token">
        <!--  plugin body will be executed if the key is not found in global cache  -->
        <var-def name="mailto">tuk.tuk.rpa@gmail.com</var-def>
        <var-def name="mailfrom">xxx@gmail.com</var-def>
        <var-def name="secure_store_alias">sz_mail_cred</var-def> 
                
        <mail
            smtp-host="smtp.yandex.com"
            smtp-port="587"
            type="html"
            to="${mailto}"
            cc="xxx@gmail.com"  
            from="${mailfrom}"
            subject="Subject"
            charset="UTF-8"
            username="${username}"
            password="${password}"
            security="ssl">
            <template>
                <![CDATA[ Please review this ]]>
                <![CDATA[ <a href="${applicationHost}/workfusion/secure/business-process/edit/${item.getWrappedObject().getRun().getRootRunUuid()}">Business Process</a>]]>
            </template>
        </mail>
        <var-def name="token">1</var-def>
    </cache>

note

Mind how the cache unique key is generated. If you hardcode it, the next run of this Business Process does NOT send an email because a cache record is found.

---

Conversion plugins

Conversion plugins are used to convert input data to a custom format.

The plugin contains the following attributes:

Name	Required	Default	Description
output-format	No		Format converted to. Depends on the conversion plugin.
on-error	No	EXCEPTION	Bot config behavior on exception during conversion: EXCEPTION throws RuntimeException. EMPTY_VALUE returns an empty string. ORIGINAL_VALUE returns the original value. DEFAULT_VALUE returns default value defined in the on-error-default-value attribute.
on-error-default-value	No		Default value for on-error is DEFAULT_VALUE.

• If on-error is specified, it works like on-error="EXCEPTION".
• If on-error="DEFAULT_VALUE" is specified and on-error-default-value is not specified, it works like on-error="EXCEPTION".

convert-date

Open plugin description

The plugin is used to convert input data to a custom date format.

If output-format is not specified, the default format MM/dd/yyyy is used.

The plugin includes the following additional attributes:

Name	Required	Default	Description
current-date	No		Define the current date to make it possible to convert dates like "yesterday" and "Monday".
for	No		Comma-separated field names that have to be converted.

Example:

<?xml version="1.0" encoding="UTF-8"?>
<config charset="UTF-8">
   
  <var-def name="date_converted">
    <convert-date output-format="MM/dd/yyyy" on-error="ORIGINAL_VALUE">
      <template>${date_check}</template>
    </convert-date>
  </var-def>
   
  <export include-original-data="true">
     <single-column name="date_converted_check" value='${date_converted}'/>
  </export>
</config>

• The input date can be without a year. In this case, the current year is used.
• If the date cannot be converted, it remains as is.

Output formats are as follows:

• MM/dd/yy
• M/d/yy
• M/d/yyyy
• MM/dd/yyyy
• MMM. d yy
• MMM dd, yy
• MMMMM dd, yy
• MMM. dd, yy
• MMM. dd yy
• MMM. d, yy
• MMM d yy
• MMM-dd-yy
• dd-MMM-yy
• MMM d, yy
• yyyy-MM-dd
• MMM-dd-yyyy
• dd-MMM-yyyy
• MMMMM dd yyyy
• MMMMM dd, yyyy
• MMM. dd, yyyy
• MMM. dd yyyy
• MMM dd yyyy
• MMM dd, yyyy
• MMM. d, yyyy
• MMM. d yyyy
• MMM d, yyyy
• MMM d yyyy
• MMMMM d
• MMM. d
• MMM d
• M/d
• MM/dd
• MM-dd
• dd-MMM
• MMM-dd
• MMM dd
• MMM. dd
• MMMMM dd

See the legend here.

ETL template example:

<?xml version="1.0" encoding="UTF-8"?>
<config charset="UTF-8">
    <var-def name="input_data">{{input_data}}</var-def>
    <var-def name="date_converted">
    <convert-date output-format="{{format}}" on-error="{{on_error}}" on-error-default-value="{{on_error_default_value}}">
        <template>${"${" + "sys.getVar(input_data.toString())" + "}"}</template>
    </convert-date>
    </var-def>
    <export include-original-data="true">
        <single-column name="{{output_data}}" value='${"${" + "date_converted" + "}"}'/>
    </export>
</config>

The plugin shows the following answers:

Answer code	Answer name	Answer type	Options	Required
input_data	Input Column Name	Free Text		Yes
output_data	Output Column Name	Free Text		Yes
date_format	Date Format	Free Text		Yes
on_error	On Error Behaviour	Select One	EXCEPTION=EXCEPTION EMPTY_VALUE=EMPTY_VALUE ORIGINAL_VALUE=ORIGINAL_VALUE DEFAULT_VALUE=DEFAULT_VALUE	Yes
on_error_default_value	On Error Default Value	Free Text		No

convert-price

Open plugin description

The plugin is used to convert input price to a custom price format.

If output-format is not specified, the C#.##. default format is used.

The plugin includes the following additional attributes:

Name	Required	Default	Description
default-currency	No	USD	If the input data is a number without currency, define the default currency with this attribute.
for	No		Comma-separated field names that have to be converted.

Example:

<?xml version="1.0" encoding="UTF-8"?>
<config charset="UTF-8">
   
  <var-def name="price_converted">
    <convert-price output-format="#.## CCCCC" on-error="ORIGINAL_VALUE">
      <template>${price_check}</template>
    </convert-price>
  </var-def>
   
  <export include-original-data="true">
     <single-column name="price_converted_check" value='${price_converted}'/>
  </export>
</config>

Output formats are as follows:

• C #.##
• C#.##
• #.##C
• #.## C
• CCC #.##
• CCC#.##
• #.##CCC
• #.## CCC
• CCCCC #.##
• CCCCC#.##
• #.##CCCCC
• #.## CCCCC
• c #.##
• c#.##
• #.##c
• #.## c
• ccccc #.##
• ccccc#.##
• #.##ccccc
• #.## ccccc
• CCC:#.##
• CCCCC:#.##
• C:#.##

See the table below for the legend:

Symbols	Description
CCCCC	Full main currency unit name, like "dollar".
CCC	Short main currency unit name, like "USD".
C	Main currency unit symbol, like "$".
ccccc	Full fractional unit name, like "cent".
c	Fractional unit symbol, like "¢".
#.##	Number format, count of sign # can be from zero to infinity.
:	Used as a delimiter.

The supported currencies are as follows:

Supported currencies	Identifier
American	Dollar, USD, $, cent, ¢
Canada	Canadian dollar, CAD, C$, cent, ¢
Euro	Euro, EUR, €, cent, ¢
United Kingdom	Pound, GBP, £, penny, pence, p, GBX
Japan	Yen, JPY, ¥
China	Yen, CNY, ¥
Switzerland	Frank, CHF, ₣

ETL template example:

<?xml version="1.0" encoding="UTF-8"?>
<config charset="UTF-8">
  <var-def name="input_data">{{input_data}}</var-def>
   
  <var-def name="price_converted">
  <convert-price output-format="{{format}}" on-error="{{on_error}}" on-error-default-value="{{on_error_default_value}}">
    <template>${"${" + "sys.getVar(input_data.toString())" + "}"}</template>
  </convert-price>
    
  </var-def>
  <export include-original-data="true">
    <single-column name="{{output_data}}" value='${"${" + "price_converted" + "}"}'/>
  </export>
</config>

The plugin shows the following answers:

Answer code	Answer name	Answer type	Options	Required
input_data	Input Column Name	Free Text		Yes
output_data	Output Column Name	Free Text		Yes
price_format	Price Format	Free Text		Yes
on_error	On Error Behaviour	Select One	EXCEPTION=EXCEPTION EMPTY_VALUE=EMPTY_VALUE ORIGINAL_VALUE=ORIGINAL_VALUE DEFAULT_VALUE=DEFAULT_VALUE	Yes
on_error_default_value	On Error Default Value	Free Text		No

convert-number

Open plugin description

The plugin is used to convert the input number to a custom number format.

The plugin includes the following additional attributes:

Name	Required	Default	Description
rounding	No	None	Rounding type: none ceil floor round
for	No		Comma-separated field names that have to be converted.

The rounding types are as follows:

Input number	none	ceil	floor	round
5.5	5.5	6	5	6
2.5	2.5	3	2	3
1.6	1.6	2	1	2
1.1	1.1	2	1	1
1.0	1.0	1	1	1
-1.0	-1.0	-1	-1	-1
-1.1	-1.1	-1	-2	-1
-1.6	-1.6	-1	-2	-2
-2.5	-2.5	-2	-3	-3
-5.5	-5.5	-5	-6	-6

If output-format is not specified, the #.## default format is used.

Example:

<?xml version="1.0" encoding="UTF-8"?>
<config charset="UTF-8">
   
  <var-def name="number_converted">
    <convert-number output-format="#,###,##0.000" on-error="ORIGINAL_VALUE" rounding="none">
      <template>${number_check}</template>
    </convert-number>
  </var-def>
   
  <export include-original-data="true">
     <single-column name="number_converted_check" value='${number_converted}'/>
  </export>
</config>

• # or 0 format means the output number is integer only.
• #. or 0 means the output always integer with the point.
• Input number 1.15 with format #.# is rounded to 1.2 on output independently of the rounding type.
• Illegal output format is zero before "number sing before" "point" like 00#.##.
• Max and min value for integer input number are 9,223,372,036,854,775,807 and 9,223,372,036,854,775,808 respectively.
• For a non-integer number, max and min value is infinity.

See the table below for the legend:

Symbols	Description
#.##	Number format.
# (before point)	Any number of digits before point (if present).
# (afer point)	One digit after point. #.### means exact three digits after the point.
0 (before or without point)	Number as is.
0 (after point)	Sets "0" digit at a specific position.
.	Decimal separator or monetary decimal separator.
,	Grouping separator.

ETL template example:

<?xml version="1.0" encoding="UTF-8"?>
<config charset="UTF-8">
  <var-def name="input_data">{{input_data}}</var-def>
   
  <var-def name="number_converted">
   
  <convert-number output-format="{{format}}" on-error="{{on_error}}" on-error-default-value="{{on_error_default_value}}" rounding="{{rounding_type}}">
    <template>${"${" + "sys.getVar(input_data.toString())" + "}"}</template>
  </convert-number>
  </var-def>
  <export include-original-data="true">
    <single-column name="{{output_data}}" value='${"${" + "number_converted" + "}"}'/>
  </export>
</config> 

The plugin shows the following answers:

Answer code	Answer name	Answer type	Options	Required
input_data	Input Column Name	Free Text		Yes
output_data	Output Column Name	Free Text		Yes
number_format	Number Format	Free Text		Yes
on_error	On Error Behaviour	Select One	EXCEPTION=EXCEPTION EMPTY_VALUE=EMPTY_VALUE ORIGINAL_VALUE=ORIGINAL_VALUE DEFAULT_VALLUE=DEFAULT_VALLUE	Yes
on_error_default_value	On Error Default Value	Free Text		No

convert-json

Open plugin description

The plugin is used to convert a JSON value of answer with the Group of Answers (GROUP) type.

The plugin contains the following attributes:

Name	Required	Default	Description
value	Yes		JSON value (multi-tab IE answer value).

The plugin takes the additional child converter attribute:

Name	Required	Default	Description
for	Yes		Comma-separated field names inside JSON that have to be converted.

Example:

<?xml version="1.0" encoding="UTF-8"?>
<config charset="UTF-8">
   
    <var-def name="json_converted" on-error="EMPTY_VALUE">
      <convert-json value="{'groupAnswerCode': [{'amount' : '100.1234567', 'score': '10', 'old_price': '1.03', 'new_price': '1.05', 'record_date': '11.12.2015'}]}">
          <convert-number output-format="#.##" on-error="EMPTY_VALUE" for="amount , "/>
          <convert-date output-format="MM/dd/yyyy" on-error="EMPTY_VALUE" for=",record_date"/>
          <convert-price output-format="#.## CCCCC" on-error="EMPTY_VALUE" for="old_price, new_price"/>
          <convert-percent on-error="EMPTY_VALUE" for="score"/>
      </convert-json>
  </var-def>
 
  <export include-original-data="true">
     <single-column name="json_converted" value='${json_converted}'/>
  </export>
</config>
----------------
json-converted - {"code":[{"amount":"100.12","score":"10%","old_price":"1.03 dollar","new_price":"1.05 dollar","record_date":"11/12/2015"}]}");

convert-percent

Open plugin description

The plugin is used to convert an input number or string to a number with the percent sign, for example, fifty-seven percent > 57%.

The plugin does not have additional attributes (only on-error and on-error-default-value).

Example:

<?xml version="1.0" encoding="UTF-8"?>
<config charset="UTF-8">
   
    <var-def name="percent_converted">
        <convert-percent on-error="ORIGINAL_VALUE">
            <template>two hundred fifty three and quarter</template>
        </convert-percent>
    </var-def>
    <!-- 253.25% -->

    <var-def name="percent_converted2">
        <convert-percent on-error="ORIGINAL_VALUE">
            <template>123 and half percent</template>
        </convert-percent>
    </var-def>
    <!-- 123.5% -->

    <var-def name="percent_converted3">
        <convert-percent on-error="ORIGINAL_VALUE">
            <template>0.7</template>
        </convert-percent>
    </var-def>
    <!-- 0.7% -->
   
    <export include-original-data="true">
        <single-column name="percent_converted" value='${percent_converted}'/>
        <single-column name="percent_converted2" value='${percent_converted2}'/>
        <single-column name="percent_converted3" value='${percent_converted3}'/>
    </export>
</config>

---

date-format

Open plugin description

The plugin allows formatting string representation of a date to the desired format. The plugin accepts dates in multiple formats and recognizes the most appropriate one.

The plugin uses the SimpleDateFormat Java class.

Date formats recognized by default are as follows:

• M/d/yyyy mm:hh:ss aa
• M/d/yy mm:hh:ss aa
• M/d/yy mm:hh:ss aa
• M/d/yy mm:hh:ss aa
• MM/dd/yy mm:hh:ss aa
• MM/dd/yyyy mm:hh:ss aa
• yy/MM/dd mm:hh:ss aa
• yyyy/MM/dd mm:hh:ss aa
• yyyy-MM-dd mm:hh:ss aa
• dd-MMM-yy mm:hh:ss aa
• dd-MMM-yyyy mm:hh:ss aa
• dd/MM/yyyy mm:hh:ss aa
• MMM-dd-yy mm:hh:ss aa
• MMM-dd-yyyy mm:hh:ss aa
• MMM dd yyyy mm:hh:ss aa
• MMM dd, yy mm:hh:ss aa
• MMMMM dd yyyy mm:hh:ss aa
• MMMMM dd, yy mm:hh:ss aa

The plugin contains the following attributes:

Name	Required	Default	Description
output-format	Yes	MM/dd/yyyy	Specifies output data format.
on-error	No	' ' (empty value)	A value to return when the date can not be parsed.
input-formats	No	See default date formats above.	A collection of acceptable date formats, can be used to provide custom formats.

Example:

<?xml version="1.0" encoding="UTF-8"?>
<config>
    
    <var-def name="date1">
      <date-format output-format="HH:mm"
             input-formats='["hh:mma"]'>
          <template>5:32AM</template>
      </date-format>
    </var-def>
    <!-- Result  05:32    -->
    
    <var-def name="date2">
      <date-format output-format="HH:mm"
             input-formats='["hh:mm a"]'>
          <template>8:17 PM</template>
      </date-format>
    </var-def>    
    <!-- Result  20:17    -->
    
    <var-def name="date3">
      <date-format output-format="HH:mm"
             input-formats='["HH:mmZ"]'>
          <template>13:00+00</template>
      </date-format>
    </var-def>
    <!-- Result  16:00    -->    
    
    <var-def name="date4">
      <date-format output-format="yyyy.MMMMM.dd hh:mm aaa z"
              input-formats='["YYYY-MM-dd HH:mm:ss"]'>
          <template>2001-07-04 12:08:56</template>
      </date-format>
    </var-def>
    <!-- Result  2001.July.04 12:08 PM MSD    -->
        
</config>

---

Excel to-from list

The excel-to-list and list-to-excel plugins read data from XLSX or XLS formats and write it to List<Map<String, Object>> [row:[columnName:value]] and vice versa.

excel-to-list

Open plugin description

The plugin must contain a child plugin, for example, http-extended or file, that returns an Excel file.

caution

Do not use excel-to-list with a standard web-harvest http plugin. Use the WorkFusion http-extended plugin. The web-harvest http plugin has a bug that causes incorrect encoding while reading binary XLS(X) files.

Example:

 <?xml version="1.0" encoding="UTF-8"?>
<config>
  <var-def name="approvers">
    <excel-to-list>
      <http-extended url="https://....xlsx"></http-extended>
    </excel-to-list>
  </var-def>
</config>

Output: For this file, after config execution, the approver variable has the following value:

Result:

<row><Exception_Approver_List>O&amp;T</Exception_Approver_List></row>
<row><Exception_Approver_List>D&apos;Onofrio, John</Exception_Approver_List></row>
<row><Exception_Approver_List>Greenbaum, Richard</Exception_Approver_List></row>
<row><Exception_Approver_List>Marcucci, Mark</Exception_Approver_List></row>
<row><Exception_Approver_List>Nerlino, Vince</Exception_Approver_List></row>
<row><Exception_Approver_List>O&apos;Leary, Terry</Exception_Approver_List></row>
<row><Exception_Approver_List>Rao, Jagdish</Exception_Approver_List></row>
<row><Exception_Approver_List>Stomberg, Sheree</Exception_Approver_List></row>

list-to-excel

Open plugin description

The plugin must contain a child node (var or template) that returns a list.

The plugin contains the following attributes:

Name	Required	Default	Description
format	No	XLSX	You can also use format="xls" to generate a file in the Excel 97 binary file format.

Output: The plugin returns a byte array.

Example:

<?xml version="1.0" encoding="UTF-8"?>

<config>
  <var-def name="peopleList">
    <template>[
       { "first_name":"John", "last_name":"Smith" },
       { "first_name":"Kevin", "last_name":"Johnson", "comment": "Optional comment" }
    ]</template>
  </var-def>  

  <file path="./output/people.xlsx" action="write" type="binary">
    <list-to-excel format="xlsx">
      <var name="peopleList"/>
    </list-to-excel>
  </file>
</config>

list-to-excel and S3 example:

<?xml version="1.0" encoding="UTF-8"?>
<config>

    <var-def name="convertedJSON">
        <script return="converted"><![CDATA[
            import java.util.HashMap;
            import java.util.ArrayList;
            import com.google.gson.Gson;

            List resultList = new ArrayList();

            Map nestedMap = new HashMap();
            nestedMap.put("one", "1");
            nestedMap.put("two", "2");
            nestedMap.put("three", "3");    

            Map anotherNestedMap = new HashMap();
            anotherNestedMap.put("one", "uno");
            anotherNestedMap.put("two", "dos");
            anotherNestedMap.put("three", "tres");  
            anotherNestedMap.put("four", "quatr");  

            resultList.add(nestedMap);
            resultList.add(anotherNestedMap);

          Gson gson = new Gson();
           String converted = gson.toJson(resultList);

        ]]></script>
    </var-def>

    <var-def name="excelFile">
        <list-to-excel format="xlsx">
            <var name="convertedJSON"/>
        </list-to-excel>
    </var-def>

    <var-def name="fileS3Location">
        <s3 bucket="temp.bucket">
            <s3-put path="my_folder/test-excel.xlsx"  content-type="application/vnd.ms-excel" content-disposition="inline" acl="PublicRead">
              <script return='excelFile'/>
            </s3-put>
        </s3>
    </var-def>

    <export include-original-data="true">
        <single-column name="file_s3_location" value="${fileS3Location.toString()}"/>
    </export>

</config>

---

excel-to-csv

Open plugin description

You can use the to-csv plugin to convert an Excel file to a CSV file.

The plugin contains the following attributes:

Name	Required	Default	Description
url	Yes (if file is missing)		URL to an Excel file (XLS or XLSX).
file	Yes (if URL is missing)		Path to an Excel file on the file system (local or server).
separator	No	;	Separator for CSV cells.

The plugin returns a text string that can be written to a file.

Example:

<?xml version="1.0" encoding="UTF-8"?>
<config>

    <var-def name="thrifts">
        <file path="./output/thrifts.csv" action="write" type="binary">
            <to-csv url="https://new-bucket-qatest.s3.amazonaws.com/user/thrifts-by-name-v1.xls"/>
        </file>
    </var-def>

    <var-def name="people">
        <file path="./output/people.csv" action="write" type="binary">
            <to-csv file="people.xlsx" separator="|"/>
        </file>
    </var-def>
    
    <export include-original-data="">
        <single-column name="thrifts" value="${thrifts}"/>
        <single-column name="people" value="${people}"/>
    </export>
</config>

---

export

Open plugin description

The plugin enables storing the data to be available on the next step (bot, manual, or business rule).

A Bot Config without the export plugin is executed constantly with no delays. To enable delays in this case, use the release plugin.

The plugin contains the following attributes:

Name	Required	Default	Description
include-original-data	Yes		Boolean value defining whether to save input data in the result snapshot or not.
export-type	No	CSV	Snapshot type: XLS for Excel until 2007 XLSX for Excel 2007 other or empty for CSV
column-name-case	No	keep	Define the case of column names in the output file: upper lower
export-columns	No		A list of columns to be added to the output file. Example: invoice_date, issuer_name, CUSIP.

Mind the following common mistakes in Bot Task design as they can lead to memory consumption issues:

• Exporting large documents between steps: The Control Tower engine stores all export results in its database and loads them for the next processing steps.

  When using a split rule, the Control Tower engine duplicates all outcoming data for each new record. For example, you have a 10 MB document and split its content into 100 records. In this case, Control Tower needs 1 GB of RAM to process the step with split data successfully.

  To avoid issues, it is recommended to save large documents or JSONs to Data Stores or S3 File Storage (if data are not changed on the next steps) and read them afterward only when needed.

• Exporting useless data between steps: The Control Tower engine stores all export results in its database and loads them for the next processing steps.

  To avoid issues, export as few columns as possible between steps and use the include-original-data attribute with the false value as much as possible. Ideally, store all the data in the Data Store and export only its id or uuid to the next step.

  When using the include-original-data attribute with the true value, you collect all data from all previous steps and what you add on the current step and store it in the Control Tower database. As a result, the same data is stored in all Business Process steps, and the DB size is growing faster, which creates additional load for the Data Purge functionality.

single-column

Open plugin description

To save one answer for one submission, use the single-column plugin.

Examples: Obtaining Alexa's rank for a website, obtaining information about a website from Whois.

The plugin contains the following attributes:

Name	Required	Default	Description
name	Yes		Name of the column in the result snapshot where the answer is to be saved.
value	No		Value to store. It is optional because you can use the body of a single-column element to obtain this value.

Examples:

<?xml version="1.0" encoding="UTF-8"?>
<config>
    <var-def name="rank">
        Rank_value
    </var-def>
 
    <export include-original-data="true">
        <single-column name="rank" value="${rank}"/>
    </export>
</config>

<?xml version="1.0" encoding="UTF-8"?>
<config>
    <var-def name="rank">
        Rank_value
    </var-def>
    <export include-original-data="true">
        <single-column name="rank">
            <var name="rank"/>  
        </single-column>
    </export>
</config>

Output of multiple variables from Bot:

<?xml version="1.0" encoding="UTF-8"?>
<config>

    <var-def name="rank">High</var-def>
    <var-def name="amount">100</var-def>
    <var-def name="title">Yahoo</var-def>

    <export include-original-data="true">
        <single-column name="rank_out" value="${rank}" />
        <single-column name="amount_out" value="${amount}" />
        <single-column name="title_out" value="${title}" /> 
    </export>


</config>

multi-column

Open plugin description

To save multiple answers for one submission, use the multi-column plugin.

Example: Searching sport club addresses by an input zip.

The plugin contains the following attributes:

Name	Required	Default	Description
list	Yes		List of objects to be stored.
split-results	No	False	Boolean value: true: the system splits data into different answers. false: the system does not split.

The multi-column plugin must contain at least one of the following child elements:

• put-to-column
• put-to-column-getter
• put-to-column-method
• put-to-plugin-method-chain

These elements determine how to access each object and can be specified explicitly or using the loop or case plugins.

If no child elements are specified, nothing is displayed for the object. More information and examples of using these plugins are provided below.

Basic example with split results:

<?xml version="1.0" encoding="UTF-8"?>
<config xmlns="http://web-harvest.sourceforge.net/schema/1.0/config">
    <script><![CDATA[
        recordsToExport = new ArrayList();
 
        Map record = new HashMap();
        record.put("column1", "value1");
        record.put("column2", "value2");
        record.put("column3", "value3");
        record.put("column4", "value4");
        record.put("column5", "value5");
     
        recordsToExport.add(record);
         
        record = new HashMap();
        record.put("column1", "value1");
        record.put("column2", "value2");
        record.put("column3", "value3");
        record.put("column4", "value4");
        record.put("column5", "value5");
     
        recordsToExport.add(record);
    ]]></script>
 
    <export include-original-data="true">
        <multi-column list="${recordsToExport}" split-results="true">
            <put-to-column-getter name="column1" property="column1" />
            <put-to-column-getter name="column2" property="column2" />
            <put-to-column-getter name="column3" property="column3" />
        </multi-column>
    </export>
</config>

Split results with a dynamic amount of output columns:

<?xml version="1.0" encoding="UTF-8"?>
<config xmlns="http://web-harvest.sourceforge.net/schema/1.0/config" scriptlang="groovy">


    <script><![CDATA[
        def amountOfRows = 100;
        def amountOfColumns = 50;
        def rand = new Random();
    
        def recordsToExport = new ArrayList();  
          
        for (i = 0; i < amountOfRows; i++) {
            Map record = new HashMap()
        
            for (j = 0; j < amountOfColumns; j++) {
                record.put("column" + j, Math.abs(rand.nextInt() % 600) + 1 )
            }
            
            recordsToExport.add(record)
        }
        
        sys.defineVariable("recordsToExport", recordsToExport)
        
        if (!recordsToExport.isEmpty()) {
            def columns = recordsToExport.get(0).keySet()
            sys.defineVariable("columns", columns)
        }
    ]]></script>
  
 
  
    <export include-original-data="false">
        <multi-column list="${recordsToExport}" split-results="true">
            <loop item="columnName">
                <list>
                    <script return="columns"/>
                </list>
                <body>
                    <put-to-column-getter name="${columnName}" property="${columnName}" />
                </body>
            </loop>
        </multi-column>
    </export>
</config>

put-to-column

Open plugin description

The plugin is intended to access the string representation of each object in a list.

The plugin contains the following attributes:

Name	Required	Default	Description
name	Yes		Name of the column in the result snapshot where the answer is to be saved.
method-chain	Yes		Methods or method chain to access the object, for example, getLanguage().getLang().

put-to-column-getter

Open plugin description

The plugin is intended to access data by the stated getter of each object in a list.

The plugin contains the following attributes:

Name	Required	Default	Description
name	Yes		Name of the column in the result snapshot where the answer is to be saved.
property	Yes		Name of the accessed object property. For example, if each object in the list has the getName method, you need to state the name property.

put-to-column-method

Open plugin description

The plugin is intended to access data using the stated method of each object in the list.

The plugin contains the following attributes:

Name	Required	Default	Description
name	Yes		The attribute stands for the column's name in the result snapshot where the answer is to be saved.
method	Yes		Name of the accessed object method. For example, if each object in the list has the obtainBalance method, state the obtainBalance method.

put-to-plugin-method-chain

Open plugin description

The plugin is intended to access data like in EL syntax—using method calls with parameters or method chain calls.

The plugin contains the following attributes:

Name	Required	Default	Description
name	Yes		Name of the column in the result snapshot where the answer is to be saved.
method-chain	Yes		Methods or method chain to access the object, for example, getLanguage().getLang().

Split JSON object into multiple records.

Examples:

<?xml version="1.0" encoding="UTF-8"?>
<config xmlns="http://web-harvest.sourceforge.net/schema/1.0/config"
    scriptlang="groovy">

    <var-def name="grid_obj">
        <json>
            {  
               "grid":[  
                  {  
                     "first_name":"alex",
                     "last_name":"zi"
                  },
                  {  
                     "first_name":"joe",
                     "last_name":"doe"
                  }
               ]
            }
        </json>
    </var-def>

    <script><![CDATA[
        num_objects = grid_obj.getWrappedObject().get(0).getWrappedObject().get("grid").size();
        parsedData = grid_obj.getWrappedObject().get(0).getWrappedObject().get("grid");
        recordsToExport = new ArrayList();
        
        for(int i = 0; i < num_objects; i++) 
        {
            record = new HashMap();
            
            item_name = parsedData.get(i).get("first_name").asText();
            item_surname = parsedData.get(i).get("last_name").asText();
            
            record.put("first_name", item_name);
            record.put("last_name", item_surname);
            
            recordsToExport.add(record);
        }
    ]]></script>

    <export include-original-data="false">
        <multi-column list="${recordsToExport}" split-results="true">
            <put-to-column-getter name="first_name" property="first_name" />
            <put-to-column-getter name="last_name" property="last_name"/>
        </multi-column>
    </export>

</config>

<?xml version="1.0" encoding="UTF-8"?>
<config>
 
    ...
 
    <export include-original-data="true">
        <multi-column list="${languages}" split-results="true">
            <put-to-plugin-method-chain name="language" method-chain="getLanguage().getLang()"/>
            <put-to-plugin-method-chain name="probability" method-chain="getLanguage().getProb()"/>
            <put-to-column-getter name="error" property="error"/>
        </multi-column>
    </export>
</config>

<?xml version="1.0" encoding="UTF-8"?>
<config>
    
    ...
 
    <export include-original-data="true">
        <multi-column list="${infos}">
            <put-to-column-getter name="created-date" property="createdDate"/>
            <put-to-column name="string_representation"/>
            <put-to-plugin-method-chain name="registrant-organization" method-chain="getRegistrant().getOrganization()"/>
            <put-to-column-method name="admin-contact-name" method="receiveContact"/>
        </multi-column>
    </export>
</config>

---

http-extended

Open plugin description

The plugin is a backport of the http plugin from the WebHarvest trunk codebase. For more details, see here.

The plugin features are as follows:

• Multiple parameters with the same name:

  <var-def name="extractResponse">
      <http-extended url="${extractUrl}" method="POST">
          <http-param-extended name="model"><template>${model}</template></http-param-extended>
          <http-param-extended name="document"><template>${document}</template></http-param-extended>
          <http-param-extended name="tag"><template>${modelOutput}</template></http-param-extended>
          <loop item="tag">
              <list>
                  <var name="tags" />
              </list>
              <body>
                  <http-param-extended name="tag"><template>${tag}</template></http-param-extended>
              </body>
          </loop>
      </http-extended>
  </var-def>

• Retries with an interval:

  <var-def name="httpResponse">
      <http-extended url="${extractUrl}" retry-attempts="5" retry-delay="6000" method="POST"/>
  </var-def>

• Standard error handling with <try> <catch>.

Standard http-param and http-header do not work with this plugin. Use http-param-extended and http-header-extended instead.

Translation API usage example:

<?xml version="1.0" encoding="UTF-8"?>
<config>

    <var-def name="translateJSON">
        {"from":"eng","to":"fra","text":"We help customers do business better by leveraging our industry-wide experience, deep technology expertise, comprehensive portfolio of services and vertically aligned business model"}
    </var-def>

    <var-def name="translateResponse">
        <http-extended url="https://lc-api.sdl.com/translate" method="POST" content-type="application/json">
            <http-header-extended name="Authorization">LC apiKey=Ur6H9PHyudrxZhqWLWjNOg%3D%3D</http-header-extended>                                 
            
            <!-- POST body is in variable below -->
            <var name="translateJSON"/>
        </http-extended>
    </var-def>
    
    <export include-original-data="true">
            <single-column name="translate_response" value="${translateResponse}"/>
    </export>    
    
</config>

By default, some headers are included with every query. All of them can be overridden with the matching http-header-extended header. You can also disable all of them by using the include-default-header=false plugin property.

Example:

// Prefer English
headers.put("Accept-Language", "en-us,en-gb,en;q=0.7,*;q=0.3");
// Prefer UTF-8
headers.put("Accept-Charset", "utf-8,ISO-8859-1;q=0.7,*;q=0.7");
// Prefer understandable formats
headers.put("Accept", "text/html,application/xml;q=0.9,application/xhtml+xml,text/xml;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5");
// Force validation of the resource in the intermediate proxies. Web browsers usually send the same header.
// Important for sites like http://skillnet.com/transactions/.
headers.put("Cache-Control", "no-cache");

http-param-extended

Open plugin description

The plugin adds the http parameter for the first enclosing HTTP processor for both post and get requests. If used outside the HTTP processor, an exception is thrown.

The syntax is as follows:

<http-param-extended name="param_name" isfile="isfile" contenttype="contenttype" filename="filename"> 
    body as parameter value 
</http-param-extended>

The plugin contains the following attributes:

Name	Required	Default	Description
name	Yes		The name of the http parameter.
isfile	No	No	Tells if the parameter is a file for upload. Applies to multipart requests only.
contenttype	No		MIME type of the upload file. Effective for multipart forms where the parameter is a file.
filename	No		Name of the uploaded file. Effective for multipart forms where the parameter is a file.

Example:

<var-def name="paramNames">
  USERID 
  PASSWORD
</var-def> 
<http-extended method="post" url="http://www.nytimes.com/auth/login"> 
    <http-param-extended name="is_continue">true</http-param-extended> 
    <http-param-extended name="URI">http://</http-param-extended> 
    <http-param-extended name="OQ"></http-param-extended> 
    <http-param-extended name="OP"></http-param-extended> 
    <loop item="name"> 
        <list> 
            <var name="paramNames"/> 
        </list> 
        <body> 
            <http-param-extended name="${name}">web-harvest</http-param-extended> 
        </body> 
    </loop> 
</http-extended> 

The plugin sends needed parameters to www.nytimes.com/auth/login to log in.

http-header-extended

Open plugin description

The plugin defines the HTTP header for the first enclosing HTTP processor. If used outside the HTTP processor, an exception is thrown.

The syntax is as follows:

<http-header-extended name="header_name"> 
    body as header value 
</http-header-extended>

The plugin contains the following attributes:

Name	Required	Default	Description
name	Yes		The name of the http header.

Example:

<http-extended url="www.google.com"> 
    <http-header-extended name="User-Agent">Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.1) Gecko/20060111 Firefox/1.5.0.1</http-header-extended> 
</http-extended>

In the example, the plugin identifies itself to www.google.com as the Firefox browser.

---

include-config

Open plugin description

The plugin is used to execute a specified WorkFusion Bot configuration in the scope of the current execution.

Recursive inclusions are supported in Control Tower but not in Eclipse, where the "Infinite loop has been found for one of the included configurations" issue occurs.

Example:

<?xml version="1.0" encoding="UTF-8"?>
<config
   xmlns="http://web-harvest.sourceforge.net/schema/1.0/config" scriptlang="groovy">

  <include-config code="UpdateDataStoreRecords" />
  <include-config code="UpdateDataStoreRecords" />

   <script><![CDATA[
      
      result = ""
      
   ]]></script> 

   <export include-original-data="true">      
   </export>
</config>

The plugin contains the following attributes:

Name	Required	Default	Description
code	Yes		Unique identifier of included config (persisted in MACHINE_CONFIG.inclusionCode). You can view the identifier of a Bot config on the Design Task tab, by default, it has the following format: 6a65fc70-c23e-45ab-ae1a-102ad4ee4269. It's a good idea to change it to something more meaningful. When developing code in Eclipse, a filename without extension plays the role of the Bot config's code.

Example:

<?xml version="1.0" encoding="UTF-8"?>
<config charset="UTF-8">
    <include-config code="bot-config-unique-identifier"/>
</config>

---

JSON manipulation plugins

The JSON plugins are intended for the following operations:

• Convert a JSON string to an object.
• Search in JSON (using XPath analog–JsonPath).
• Delete JSON nodes.
• Change node values.
• Add new JSON nodes.

reference

• ObjectNode class API
• Use API link for updated Jackson
• JsonPath SDL description
• Online evaluator for testing a JSON expression
• JSON expression manual with examples from Stefan Gössner

The plugin has the following common attribute:

Name	Required	Description
expression	No	JsonPath expression to find a key in the JSON. Examples: "$.response.roles" "$.store.book[?(@.author == 'John Snow')]"

json

Open plugin description

The plugin converts a JSON string to an ObjectNode object and can contain other JSON plugins inside itself.

Included config with the store_json and new_json_object variables:

<?xml version="1.0" encoding="UTF-8"?>
<config>
 
    <var-def name="new_json_object">
        <json>["element_1", "element_2", "element_3"]</json>
    </var-def>

    <var-def name="store_json">
        <template>
            {
                "store": {
                    "book": [
                        {
                            "category": "reference",
                            "author": "Nigel Rees",
                            "title": "Sayings of the Century",
                            "price": 8.95
                        },
                        {
                            "category": "fiction",
                            "author": "Evelyn Waugh",
                            "title": "Sword of Honour",
                            "price": 12.99
                        },
                        {
                            "category": "fiction",
                            "author": "Herman Melville",
                            "title": "Moby Dick",
                            "isbn": "0-553-21311-3",
                            "price": 8.99
                        },
                        {
                            "category": "fiction",
                            "author": "J. R. R. Tolkien",
                            "title": "The Lord of the Rings",
                            "isbn": "0-395-19395-8",
                            "price": 22.99
                        }
                    ],
                    "bicycle": {
                        "color": "red",
                        "price": 19.95
                    }
                },
                "expensive": 10
            }
        </template>
    </var-def>
</config>

JSON simple usage example:

<?xml version="1.0" encoding="UTF-8"?>
<config>

    <!-- convert JSON string to object and test that it can be converted back to the same JSON -->
    <var-def name="new_json_object">
        <json>{"color":"Black", "size":"Big"}</json>
    </var-def>

    <!-- getting object value by a key   -->
    <script><![CDATA[
      System.out.println(new_json_object.getWrappedObject().get(0).getWrappedObject().get("color"));
    ]]></script>
    <!-- "Black"   -->

</config>

Expression example:

<?xml version="1.0" encoding="UTF-8"?>
<config>

    <include-config code="store_json"/>

    <!-- extract books with price greater than 10 -->
    <var-def name="expensive_books">
        <json expression="$..book[?(@.price > 10)]">
            <var name="store_json"/>
        </json>
    </var-def>

    <export include-original-data="true">
        <single-column name="expensive_books" value="${expensive_books}"/>
    </export>

</config>

json-put

Open plugin description

The plugin creates a JSON node with a defined key and value. The value is set inside the json-put plugin.

The plugin contains the following attributes:

Name	Required	Description
key	Yes	JSON key.

Example:

<?xml version="1.0" encoding="UTF-8"?>
<config>
    <var-def name="rest_response">
        <json>
            <json-put key="status">SUCCESS</json-put>
            <json-put key="response"><json>{}</json></json-put>
            <json-put expression="$.response" key="id">25</json-put>
            <json-put expression="$.response" key="username">John Doe</json-put>
        </json>
    </var-def>
    <!-- {"status":"SUCCESS","response":{"id":"25","username":"John Doe"}}  -->
</config>

json-set

Open plugin description

The plugin sets a value to the JSON node found by expression.

Example:

<?xml version="1.0" encoding="UTF-8"?>
<config>
    <include-config code="store_json"/>
    <var-def name="initial_json">
        <var name="store_json"/>
    </var-def>
    
    <var-def name="updated_books">
        <json>
            <var name="store_json"/>
            <json-set expression="$.store.book[?(@.price &gt; 10)].price">5.77</json-set>
        </json>
    </var-def>
</config>

json-add

Open plugin description

The plugin adds a new node to the parent JSON.

Example:

<?xml version="1.0" encoding="UTF-8"?>
<config>
    <var-def name="rest_response">
        <json>
            <json-put key="status">SUCCESS</json-put>
            <json-put key="response"><json>{}</json></json-put>
            <json-put expression="$.response" key="id">25</json-put>
            <json-put expression="$.response" key="username">John Doe</json-put>
            <json-put expression="$.response" key="roles"><json>[]</json></json-put>
            <json-add expression="$.response.roles">ROLE_ADMIN</json-add>
            <json-add expression="$.response.roles">ROLE_POWERUSER</json-add>
            <json-add expression="$.response.roles">ROLE_DATASTORE</json-add>
        </json>
    </var-def>
    <!-- {"status":"SUCCESS","response":{"id":"25","username":"John Doe","roles":["ROLE_ADMIN","ROLE_POWERUSER","ROLE_DATASTORE"]}} -->
    
 
    <var-def name="dynamic_json">
        <json>
            []
            <while condition="true" maxloops="3">
                <json-add>
                    <json>
                        <json-put key="first_name">Agent</json-put>
                        <json-put key="last_name">Smith</json-put>
                    </json>
                </json-add>
            </while>
        </json>
    </var-def>
    <!-- [{"first_name":"Agent","last_name":"Smith"},{"first_name":"Agent","last_name":"Smith"},{"first_name":"Agent","last_name":"Smith"}]  -->
</config>

json-delete

Open plugin description

The plugin deletes nodes that match an expression of the parent json plugin. The plugin structure is as follows:

<json expression="expression_to_search_in_child_json">
    <var name="json_to_delete_found_nodes_from"/>
    <json-delete />
</json>

Example:

<?xml version="1.0" encoding="UTF-8"?>
<config>
    <include-config code="store_json"/>
    <var-def name="initial_json">
        <var name="store_json"/>
    </var-def>
    <var-def name="deleted_json">
        <json expression="$.store.book[?(@.price != 22.99)]">
            <var name="store_json"/>
            <json-delete />
        </json>
    </var-def>
    <!-- {"store":{"book":[{"category":"fiction","author":"J. R. R. Tolkien","title":"The Lord of the Rings","isbn":"0-395-19395-8","price":22.99}],
    "bicycle":{"color":"red","price":19.95}},"expensive":10}   -->
</config>

Complex JSON example:

<?xml version="1.0" encoding="UTF-8"?>
<config>
    <include-config code="store_json"/>
    
    <!-- add new attribute and put "new_json_object" inside -->
    <var-def name="updated_books">
        <json>
            <var name="store_json"/>
            <json-put key="abc">Just String Value</json-put>
            <json-put key="abcabc1">Just String Value</json-put>
        </json>
    </var-def>
    <!-- it is possible to put complex objects -->
    <var-def name="updated_books_1">
        <json>
            <var name="store_json"/>
            <json-put expression="$.store.book[?(@.price &gt; 10)]" key="category"><var name="new_json_object" /></json-put>
        </json>
    </var-def>
    <!-- or objects created in Java -->
    <var-def name="newValue">
        <script return="newMap"><![CDATA[
            Map newMap = new HashMap();
            newMap.put("key1", "val1");
            newMap.put("key2", "val2");
        ]]></script>
    </var-def>
    <var-def name="updated_books_2">
        <json expression="$.store.book[?(@.author == 'Pavel Valetka')]" >
            <var name="store_json"/>
            <json-put key="new_attributes"><var name="newValue" /></json-put>
        </json>
    </var-def>
    <!-- delete nodes matches expression -->
    <var-def name="updated_books_3">
        <json expression="$.store.book[?(@.price &gt; 10)]">
            <var name="updated_books_1"/>
            <json-delete />
        </json>
    </var-def>
    <var-def name="new_books">
        <script return="newBooks"><![CDATA[
            List newBooks = new ArrayList();
            Map book1 = new HashMap();
            book1.put("category", "fantasy");
            book1.put("author", "Ivanov Ivan");
            book1.put("title", "Very Interesting Book");
            book1.put("price", "1000000.00");
            Map book2 = new HashMap();
            book2.put("category", "fantasy");
            book2.put("author", "Petrov Petr");
            book2.put("title", "Boring Book");
            book2.put("price", "3.00");
            newBooks.add(book1);
            newBooks.add(book2);
        ]]></script>
    </var-def>
    <var-def name="updated_books_4">
        <json expression="$.store.book">
            {
            "store": {
            "book": [
            {
            "category": "reference",
            "author": "${dynamic_author}",
            "title": "Sayings of the Century",
            "price": "8.95"
            },
            {
            "category": "fiction",
            "author": "Evelyn Waugh",
            "title": "Sword of Honour",
            "price": "12.99"
            }
            ],
            "bicycle": {
            "color": "red",
            "price": "19.95"
            }
            }
            }
            <json-add><var name="new_books"/></json-add>
        </json>
    </var-def>
    <var-def name="updated_books_5">
        <json expression="$.store.book">
            <json-add><var name="new_books"/></json-add>
            <var name="store_json"/>
        </json>
    </var-def>
    <var-def name="updated_books_6">
        <json expression="$.store.book[?(@.price &lt; 10)].price">
            <json-set>5.77</json-set>
            <var name="updated_books_4"/>
        </json>
    </var-def>

    <var-def name="updated_books_6">
        <json>
            <json-set expression="$.store.book[?(@.price &lt; 10)].price">5.77</json-set>
            <var name="updated_books_4"/>
        </json>
    </var-def>

    <!--
        Construct totally new object
        $ - default expression
    -->
    <var-def name="rest_response">
        <json>
            <json-put key="status">SUCCESS</json-put>
            <json-put key="response">
                <json>
                    <json-put key="id">25</json-put>
                    <json-put key="username">Pavel Valetka</json-put>
                    <json-put key="roles">
                        <json>
                            <json-add>ROLE_ADMIN</json-add>
                            <json-add>ROLE_POWERUSER</json-add>
                            <json-add>ROLE_DATASTORE</json-add>
                            []
                        </json>
                    </json-put>
                    {}
                </json>
            </json-put>
            {}
        </json>
    </var-def>
    
</config>

---

language-extractor

Open plugin description

The plugin is used to detect the language of the given website.

The plugin contains the following attributes:

Name	Required	Default	Description
url	Yes		The URL of the website to detect the language.

The plugin returns a list containing instance(s) of the com.freedomoss.crowdcontrol.webharvest.plugin.langextractor.LanguageResultItem class.

You can use the following methods to get the Language and its Probability:

• getLanguage().getLang()
• getLanguage().getProb()

Example:

<?xml version="1.0" encoding="UTF-8"?>
<config>
    <var-def name="language">
        <language-extractor url="${url}"/>
    </var-def>
    <export include-original-data="true">
        <multi-column list="${language}">
            <put-to-plugin-method-chain name="language" method-chain="getLanguage().getLang()"/>
            <put-to-plugin-method-chain name="probability" method-chain="getLanguage().getProb()"/>
            <put-to-column-getter name="error" property="error"/>
        </multi-column>
    </export>
</config>

---

list-to-csv

Open plugin description

The plugin can be used to export JSON-formatted data to a given CSV file.

The plugin contains the following attributes:

Name	Required	Default	Description
separator	No	, (comma)	Separator for CSV cells.
quote-symbol	No	No symbol	Quote symbol.
escape-symbol	No	No symbol	Symbol for escaping special characters. With the default value, it still escapes characters that can break CSV structure, like commas in case of a comma separator.

Use the list of simple JSON objects as input:

[
       { "key":"value", ...},
        ...
]

The plugin returns a byte array.

Example:

<?xml version="1.0" encoding="UTF-8"?>
 
<config>
  <var-def name="cities">
    <template>[
       { "city":"Minsk", "estimated_people_count": 2100000, "post_code": "220000" },
       { "city":"Krakow", "estimated_people_count": 780000 }
    ]</template>
  </var-def> 
     
  <file path="./output/cities.csv" action="write" type="binary">
    <list-to-csv separator="|">
      <var name="cities"/>
    </list-to-csv>
  </file>
</config>

---

log

Open plugin description

You can view the plugin log messages on View > Results > Summary > the Run's events popup and in the log file.

By default, WARN and higher messages appear on the events view. You can change the configuration using standard logback.

The plugin contains the following attributes:

Name	Required	Default	Description
message	No		Logging message, message can be set in body. Body's value has more priority than attribute value. The message limit is 1000 characters. If the limit is exceeded, only the first 1000 characters are saved.
level	No		Logging level, possible values: ERROR, WARN, INFO, DEBUG, TRACE, default INFO. For events, view the DEBUG and TRACE logging level equal to INFO.

Example:

<?xml version="1.0" encoding="UTF-8"?>
<config>
    <log message="attribute message, default level (INFO)"/>
    <log message="attribute message, level=error" level="ERROR"/>
    <log message="attribute message, level=warn" level="WARN"/>
    <log message="attribute message, level=info" level="INFO"/>

    <var-def name="messageToLog">
        custom message details
    </var-def>
    <log>
        <template>body message, level=default. Message details: ${messageToLog}</template>
    </log>
     
    <log level="WARN">
        body message level=default
    </log>
 
    <script>
          <![CDATA[
          log.info("logs info message. Message details: {}", messageToLog.toString());
          log.warn("logs warn message");

          try {
              Integer i = null;
              i.intValue();
          } catch (Exception e) {
              log.error("Error!!!", e);
              log.error("Message details: {}. Error short: {}", messageToLog.toString(), e.getMessage());
              log.debug("Error debug!!!");
          }
       ]]>
    </script>
</config>

---

mail-check

Open plugin description

The plugin is intended for connecting to a mail server and checking for new unread emails with a specified subject pattern.

If there are some new emails matching all defined criteria, the plugin returns a list of these emails with a brief description of each like from, subject, body, priority, size, attachments.

Output:

Email{from='John Smith <jsmith@example.com>', subject='SSI docs', body='Here are docs...', priority=normal, size=53257, attachments=[EmailAttachment{fileName='SSIDocument_AUD.PDF', contentType='application/pdf', size=48094, data(size)=35144}]}
Email{from='Bjorn <bjorn@example.com>', subject='New SSI', body='New documents..', priority=normal, size=6228, attachments=[]}

The plugin contains the following attributes:

Name	Required	Default	Description
connection-props	Yes		A map containing protocol, host, port, and security.
user	Yes		Your username (email address).
password	Yes		Password. You can use Secrets Vault plugins to store username and password.
folder	No	INBOX	A mail box folder to check the messages.
subject-pattern	No	any string	Only emails that contain this string in their subject are included in the config output.
max-messages	No	10	Maximal number of emails in the result.
max-message-size	No	20971520	Only emails that do not exceed this size are included in the config output.

For the Data Store with input parameters, see the table below:

Key	Value
mail.store.protocol	IMAPS
mail.host	your_host
mail.imaps.port	993
mail.user.name	email@example.com
mail.user.password	super_secure_password
subject-pattern	Invoice
max-messages	5
max-message-size	0
mail.smtp.host	your.smtp.host
mail.smtp.port	465
mail.smtp.security	SSL
mail.notification.reply	email.reply@example.com

Example:

<?xml version="1.0" encoding="UTF-8"?>
<config>
    <script>
        <![CDATA[
            Map initParamsMap = new HashMap();
            java.util.concurrent.TimeUnit.SECONDS.sleep(15);
        ]]>
    </script>

    <loop item="initParam">
        <list><datastore name="mailcheck_settings">select * from @this;</datastore></list>
        <body>
            <template>${initParamsMap.put(initParam.get("key").toString(),initParam.get("value").toString())}</template>
        </body>
    </loop>

    <script>
        <![CDATA[
            Properties connectionProps = new Properties();

            for (Object obj : initParamsMap.entrySet()){
                Map.Entry pair = (Map.Entry) obj;
                String k = pair.getKey().toString();
                if (k.startsWith("mail.")) {
                    connectionProps.put(k, pair.getValue().toString());
                }
            }
        ]]>
    </script>

    <var-def name="new_mails">
        <mail-check
            connection-props="${connectionProps}"
            user='${initParamsMap.get("mail.user.name")}'
            password='${initParamsMap.get("mail.user.password")}'
            subject-pattern='${initParamsMap.get("subject-pattern")}'
            max-messages='${initParamsMap.get("max-messages")}'
            max-message-size='${initParamsMap.get("max-message-size")}'
        />
    </var-def>

    <export include-original-data="true">
        <single-column name="new_mails" value="${new_mails}"/>
    </export>
    
</config>

---

pool

Open plugin description

Initially, the pool plugin was created to provide exclusive access to a record from a Data Store with external system credentials.

For example, one user can use only one thread simultaneously. In the current implementation, it is generalized to any object list.

The plugin provides the semaphore functionality for objects in the list section. For each invocation, the plugin returns any not borrowed object from the evaluated list. Check whether the object is borrowed in a pairwise comparison based on the borrowed object equals method.

For example, for converted to Map objects, it measures the equality of all entries. By default, objects are converted to Map<String, Object>:

• DbRowVariable is based on values.

• NodeVariable is based on the wrapped object value.

• Map is passed as it is.

• POJOs are converted using BeanMap.

• If the object list is empty, an exception is thrown.

• If the pool plugin is not invoked for more than 10 minutes, all objects in the pool become accessible again.

• The pool plugin does not support the external classes loaded from MCB, if the following attribute is set convert-to-map=false. To fix, set the webharvest.config.pool.plugin.distributed=false property in the workfusion.properties file.

The pool plugin has two mandatory sections:

• list contains expression evaluated to ListVariable. Can be evaluated multiple times based on the timeout settings.
• body contains logic using the borrowed object.

The plugin contains the following attributes:

Name	Required	Description
key	Yes	Name of the pool. Each request to the pool retrieves an object not currently borrowed. Check whether the borrowed object is equal-based. If the datastore plugin is used in the list section, it is recommended to have a key equal to the Data Store name used.
item	No	Variable name where the result is recorder. Default value: poolableObject.
convert-to-map	No	Defines whether listed objects need to be converted to Map. DbRowVariable is converted based on the column name and column value. Map is left as it is. Other objects are converted using BeanMap with the removed class attribute to work with dynamic classes. Default value: true.
polling-interval	No	Value in milliseconds, negative means no limitation. It is used when there are more threads than available objects in the pool, and the pool is modified. It sets maximum wait for object borrow before retry. For example, when there are two objects in the pool and five threads. Three threads are waiting for the object with the list value already resolved. If the waiting period is too long, their list can become obsolete. After max-wait milliseconds, the list is re-read, and the borrow is repeated. Default: -1.
wait-timeout	No	Value in milliseconds. If it is exceeded in the borrow object part, PluginException is thrown. Default value: 120,000 (2 minutes).

Plugin usage on Data Store content:

<pool key="datastore" item="user">
    <list>
        <datastore name="user pool">
            select * from @this;
        </datastore>
    </list>
    <body>
        <script>
            <![CDATA[
                String login = user.get("userid");
                String pass = user.get("password");
            ]]>
        </script>
    </body>
</pool>

Plugin usage on simple POJO list:

<script>
    <![CDATA[
        List users = new ArrayList();
        users.add(new TestUser("u1", "p1", 12));
        users.add(new TestUser("u2", "p2", 18));
    ]]>
</script> 

<pool key="list" item="user2">
    <list>
        <script return="users" />
    </list>
    <body>
        <script>
            <![CDATA[
                String login2 = user2.get("name");
                String pass2 = user2.get("password");
                int age = user2.get("age");
                logger.debug("User block2 : {}", login2);
            ]]>
        </script>
    </body>
</pool>

Example with conversion turned off:

<script>
        <![CDATA[
            import org.slf4j.Logger;
            import org.slf4j.LoggerFactory;
            import org.apache.commons.lang3.tuple.Pair;
            Logger logger = LoggerFactory.getLogger("com.freedomoss.crowdcontrol.webharvest.OBJECT_POOL");
            List users = new ArrayList();
            users.add(Pair.of("u1", "p1"));
            users.add(Pair.of("u2", "p2"));
            users.add(Pair.of("u3", "p3"));
        ]]>
</script>
<pool key="convertedList" item="testUser" convert-to-map="false">
    <list>
        <script return="users" />
    </list>
    <body>
        <script>
            <![CDATA[
                String testUserName = testUser.getKey();
                String testUserPassword = testUser.getValue();
                logger.debug("TestUser : {}", testUserName);
                java.util.Random r = new java.util.Random();
                Thread.sleep(r.nextInt(5000));
            ]]>
        </script>
    </body>
</pool>

Example with timeouts:

The list is reevaluated each second until the object is borrowed from the pool. If after five minutes, the object is still not borrowed, an exception is thrown.

<?xml version="1.0" encoding="UTF-8"?>
<config
xmlns="http://web-harvest.sourceforge.net/schema/1.0/config"
scriptlang="groovy">
 
    <script>
        <![CDATA[
            import org.apache.commons.lang3.tuple.Pair;
            List users = new ArrayList();
            users.add(Pair.of("u1", "p1"));
            users.add(Pair.of("u2", "p2"));
            users.add(Pair.of("u3", "p3"));
            println " List of users "
            println users
            sys.defineVariable("users", users);
        ]]>
    </script>
    <pool key="convertedList" item="testUser" convert-to-map="false">
        <list>
            <script return="users" />
        </list>
        <body>
            <script>
                <![CDATA[
                    String testUserName = testUser.getKey();
                    String testUserPassword = testUser.getValue();
                    println " User name add password "
                    println testUserName
                    println testUserPassword
                    java.util.Random r = new java.util.Random();
                    Thread.sleep(r.nextInt(5000));
                ]]>
            </script>
        </body>
    </pool>
    
    <export include-original-data="true"></export>
</config>

---

release

Open plugin description

The plugin is used to postpone the next execution of the current record.

There are a few limitations to be considered when using the release plugin:

• Works only if the export plugin is not invoked. This limitation also applies to all included configs. Make sure that your included config does not have the export plugin block.
• The release plugin does not work with an interval of less than 60 s. The pattern is as follows:

time-in-seconds	Actual interval
1–59	60
60-119	120
120-179	180

The plugin contains the following attributes:

Name	Required	Default	Description
time-in-seconds	Yes		Minimal number of seconds HIT execution is postponed for. It's minimal, not the exact value.

You can use the release plugin for remote jobs with a long execution time:

• Submit tasks to such a job from Bot config and provide HIT UUID for callback.
• When the job is finished, it invokes the REST API to wake up HIT by UUID.
• In some cases as a replacement for Thread.sleep, which is not efficient in Bot configs. See the OCR example.

The examples are as follows:

• Automation training.
• OCRing of a large document. It can be used instead of Thread.sleep during the OCR processing status query, which is not efficient. The idea is to have one config for pre-processing and OCR task posting and another config to query OCR task status periodically.

Example:

<?xml version="1.0" encoding="UTF-8"?>
<config xmlns="http://web-harvest.sourceforge.net/schema/1.0/config" scriptlang="groovy">

    <var-def name="stop">
        false
    </var-def>

    <case>
        <if condition="${stop.toBoolean()}">
            <export include-original-data="true">
                <single-column name="stopped" value="true"/>
            </export>
        </if>
        <else>
            <script>
                <![CDATA[
                    import com.freedomoss.crowdcontrol.webharvest.WebHarvestTaskItem;
                    import com.freedomoss.crowdcontrol.webharvest.AwsHitDto;

                    AwsHitDto hit = ((WebHarvestTaskItem) item.getWrappedObject()).getSubmission().getAwsHit();
                    String hitUuid = hit.getUuid();
                    log.debug("Hit {} execution is postponed for 3 mins", hitUuid);
                ]]>
            </script>
            <release time-in-seconds="180"/>
        </else>
    </case>

</config>

tip

To execute the current record before its release timeout, you call the following API operation using hitUuid of the current record: /rest/api/hit/resume/${hitUuid}

---

required

Open plugin description

The plugin declares column names and variables to be passed in the input data file:

<Required name="Required_column_name"/>

If these columns are missing:

• In WorkFusion, a warning is displayed when saving a process: "Warning! The operation you have selected does not correspond to some columns in the upload file". In this case, you are prompted to map input columns.
• In Eclipse, an error occurs: "The following fields are Required but were not defined: list of fields".

---

script-var

Open plugin description

The plugin can be alternatively used in scripts instead of the var-def plugin to facilitate coding to access the resulting value:

• script-var:

  theVar

• var-def:

  theVar.get(0).wrappedObject() or theVar.getWrappedObject().get(0)

The syntax is as follows:

<script-var name="varName" return="resultExpression">
  body to be bound to the variable
</script-var>

The plugin contains the following attributes:

Name	Required	Description
name	Yes	The name of variable. Should be valid like in most programming languages.
return	No	Specifies what this processor should evaluate at the end and return as a processing value. It works in the same way as the script processor return attribute.

Example of using with unzip plugin:

<?xml version="1.0" encoding="UTF-8"?>
<config xmlns="http://web-harvest.sourceforge.net/schema/1.0/config"
    scriptlang="groovy">

    <script-var name="unzipped_1">
        <unzip>
            <http url="https://pub_demo.s3.amazonaws.com/trainings/split.zip"></http>
        </unzip>
    </script-var>

     <var-def name = "unzipped_2">
        <unzip>
            <http url="https://pub_demo.s3.amazonaws.com/trainings/folder.zip"></http>
        </unzip>
     </var-def>

    <script><![CDATA[

       // getting file name from archive (declared with script-var)

          unzip_directory = unzipped_1;
          file_name = unzip_directory.getChildren().get(0).getName();

        // getting files string content from a folder in archive (declared with var-def)

        dir_content = "";
        unzip_directory = unzipped_2.get(0).getWrappedObject().getChildren().get(0);
        unzip_directory.getChildren().each { filecontent ->
            dir_content += "${filecontent.toString()}"
        };

    ]]></script>


    <export include-original-data="true">
        <single-column name="file_name" value="${file_name}"/>
        <single-column name="dir_content" value="${dir_content}"/>
    </export>

Example of using with json plugin:

<?xml version="1.0" encoding="UTF-8"?>
<config xmlns="http://web-harvest.sourceforge.net/schema/1.0/config">

    <!-- convert JSON string to object and test that it can be converted back to the same JSON -->
    <script-var name="new_json_object">
        <json>{"color":"Black", "size":"Big"}</json>
    </script-var>

    <!-- getting object value by a key   -->
    <script><![CDATA[
      System.out.println(new_json_object.get("color"));
    ]]></script>
    <!-- "Black"   -->

</config>

---

similarity-score

Open plugin description

The plugin is intended for comparing two text strings and providing their similarity score. As an output, the similarity-score plugin returns a double number, for example: 0.27 or 40.0.

To understand how the score is calculated, see the String metric article.

The plugin contains the following attributes:

Name	Required	Default	Description
content1	No		First string to compare.
content2	No		String to compare with content1.
distance	No	ScaledLevenstein It operates between two input strings, returning a number equivalent to the number of substitutions and deletions needed to transform one input string into another.	Scoring algorithm (string distance calculation). For example: com.wcohen.ss.SmithWaterman com.wcohen.ss.NeedlemanWunsch

Example:

<?xml version="1.0" encoding="UTF-8"?>
<config>

    <similarity-score content1="13-339 M&amp;J RD, Myakka City, FL 34251" content2="13339 MJ RD MYAKKA CITY    FL 34251"/>
    <!-- 1.0 -->
 
    <similarity-score content1="13-339 M&amp;J RD, Myakka City, FL 34251" content2="13339 MJ RD MYAKKA CITY    FL 34251" distance="com.wcohen.ss.SmithWaterman"/>
    <!-- 52.0 -->
</config>

---

task-start

Open plugin description

Using the plugin, you can start a Manual Task or Business Process from a specific Business Process with new input data or depending on your current Business Process results.

This capability eliminates the need to upload CCF and CSV files to an external server, for example, Amazon S3, and then start a new scheduled Task or Business Process.

caution

A single bot step can't be started with this plugin.

The plugin contains the following attributes:

Name	Required	Default	Description
campaign-uuid	Yes		Business Process definition UUID (campaign uuid) or Manual Task UUID.
main-data	Yes		Content of a main data (the file content).
qualification-run	No		Boolean flag if the task is a qualification. Data for the qualification task should be provided via the main-data attribute.
qualification-training	No		Boolean flag if the task is in the training mode. The flag is ignored if qualification-run is not true.
file-separator	No		Appropriate separator used within the input data file. Valid separators include: “,” comma “;” semicolon “|” pipe “\t“ tab For example: file-separator="|"
block-size	No	0	Defines the number of records to display in one single Worker Task.
skip-bad-lines	No	False	Boolean, if true, skips lines that cannot be parsed.
task-display-priority	No		Run display priority. Should be a priority name, for example, low, normal, max.
due-date	No		ISO formatted date time. Tasks will expire after that date.
encoding	No	UTF-8	File encoding.
tags	No		User-defined tags for the Run.
expiration-data	No		Expiration data, JSON format.
custom-attributes	No		Custom attributes that should be applied on a triggered run. String in format: ["key1": "value1", "key2": "value2"]
stream-type	No		Type of streaming. Possible value PermanentOpenTask. Any other or absent value is considered as Immediately.
stream-value	No		Possible value: positive Long. Used with the PermanentOpenTask type only.
stream-threshold	No		Minimum task threshold Required to initialize streaming. Possible values: integer (absolute task number) or percent (relative), for example, 1 or 10%.
automation-training	No	False	If true, an automation Business Process starts in the training mode.

Example 1:

<?xml version="1.0" encoding="UTF-8"?>
<config>
    <script>
        <![CDATA[
          String mainData = "id, text\n1,sentiment1";
         ]]>
    </script>
    <var-def name="startTaskResult">
        <task-start
                campaign-uuid="d51cd012-bc23-4daa-a062-2a5b976e93b8"
                main-data="${mainData}" />

    <!--<task-start campaign-uuid="fail" main-data="${mainData}"/> -->
    </var-def>
    <export include-original-data="true">
        <single-column name="startTaskResult" value="${startTaskResult}"/>
    </export>
</config>

Example 2:

<?xml version="1.0" encoding="UTF-8"?>
<config>
    <script>
        <![CDATA[
          String mainData = "id, text\n1,sentiment1";
         ]]>
    </script>
    <var-def name="startTaskResult">
        <task-start
                campaign-uuid="d51cd012-bc23-4daa-a062-2a5b976e93b8"
                main-data="${mainData}"
                qualification-run="false"
                file-separator=","
                block-size="2"
                skip-bad-lines="true"
                task-display-priority="70"
                due-date="2016-09-20T19:31:36Z"
                encoding="UTF-8"
                tags="tag1,tag2"
                custom-attributes="['key1':'value1']"
                stream-type="Immediately"
                stream-value="5"
                stream-threshold="3" />

    <!--<task-start campaign-uuid="fail" main-data="${mainData}"/> -->
    </var-def>
    <export include-original-data="true">
        <single-column name="startTaskResult" value="${startTaskResult}"/>
    </export>
</config>

Result example:

//Success
{  
   "valid":true,
   "errors":[],
   "output":{  
      "startedDate":"2015-07-23T11:20:32.888Z",
      "uuid":"c7460423-eac8-4424-85c8-058406cb5a87"
   }
}

//Error
{  
   "valid":false,
   "errors":[  
      "Incorrect 'campaignUuid' parameter"
   ],
   "output":{}
}

---

to-text

Open plugin description

The plugin utilizes the Apache Tika v3.1.0 toolkit to parse the content and extract the text. See the list of Tika supported parsers at https://tika.apache.org/3.1.0/formats.html.

The plugin contains the following attributes:

Name	Required	Default	Description
parser	No		Fully qualified name of the class implementing org.apache.tika.parser.Parser that is utilized to parse the content.
handler	No		Fully qualified name of the class implementing org.xml.sax.ContentHandler that is utilized to parse the content.
extractor	No		Fully qualified name of the class implementing de.l3s.boilerpipe.extractors.ExtractorBase is utilized to extract additional information. Useful when you need to extract an article from a web page.

Basic example:

<!-- Note how we extract the text from the zipped text file stored at s3. It can be any file format supported by Apache Tika -->
<var-def name="text"> 
    <to-text>
        <http url="https://s3.amazonaws.com/temp_bucket/randomtext.txt.zip"/>
    </to-text>
</var-def>

Example of news article extraction:

<var-def name="text"> 
    <to-text parser="org.apache.tika.parser.html.HtmlParser"
             handler="org.apache.tika.parser.html.BoilerpipeContentHandler"
             extractor="de.l3s.boilerpipe.extractors.ArticleExtractor">
                    <http url="http://crowdcomputingblog.com/2014/01/22/buying-the-haystack-for-the-needle-and-4-other-common-financial-data-industry-obstacles/"/>
    </to-text>
</var-def>

---

unzip

Open plugin description

The plugin is generally used to extract the content of zip files. The plugin accepts the byte array as a source in the definition body. It can be used in conjunction with any plugins that produce byte[ ], for example, file and http.

The unzip plugin returns the FileEntity object as a recursive structure. For more details, refer to JavaDoc.

Java example:

package com.freedomoss.crowdcontrol.webharvest.plugin.zip.dto;

public class FileEntity {
    private String name;
    private FileType type;
    private byte[] content;
    private List<FileEntity> children;
    private String mimeType;

    public List<FileEntity> getChildren() {
        return children;
    }

    public byte[] getContent() {
        return content;
    }

    public String getMimeType() {
        return mimeType;
    }

    public String getName() {
        return name;
    }

    public FileType getType() {
        return type;
    }

    public boolean isDirectory() {
        return FileType.DIRECTORY == type;
    }

    public boolean isFile() {
        return FileType.FILE == type;
    }
}

Getting file names and content from archive:

<?xml version="1.0" encoding="UTF-8"?>
<config xmlns="http://web-harvest.sourceforge.net/schema/1.0/config"
    scriptlang="groovy">

    <var-def name="unzipped_1">
        <unzip>
            <http url="https://pub_demo.s3.amazonaws.com/trainings/split.zip"></http>
        </unzip>
    </var-def>

    <var-def name="unzipped_2">
        <unzip>
            <http url="https://pub_demo.s3.amazonaws.com/trainings/folder.zip"></http>
        </unzip>
    </var-def>

    <script><![CDATA[

       // getting file name from archive

        unzip_directory = unzipped_1.get(0).getWrappedObject();
        file_name = unzip_directory.getChildren().get(0).getName();

        // getting files string content from a folder in archive

        dir_content = "";
        unzip_directory = unzipped_2.get(0).getWrappedObject().getChildren().get(0);
        unzip_directory.getChildren().each { filecontent ->
            dir_content += "${filecontent.toString()}"
        };

    ]]></script>


    <export include-original-data="true">
        <single-column name="file_name" value="${file_name}"/>
        <single-column name="dir_content" value="${dir_content}"/>
    </export>

</config>

Example:

<?xml version="1.0" encoding="UTF-8"?>                
<config>
    <script>
        <![CDATA[
            fileName = this.getClass().getClassLoader().getResource("files/zip-test.zip").getPath();
        ]]>
    </script>

    <var-def name="result">
        <unzip>
            <file type="binary" path="${fileName}"/>
        </unzip>
    </var-def>
</config>

---

s3-put

Open plugin description

The plugin can accept a byte[ ] or the FileEntity structure in the body. In the second case, all files are extracted and placed in a S3 bucket. Empty extracted folders are not created on S3.

Example:

<?xml version="1.0" encoding="UTF-8"?>                
<config>
    <script>
        <![CDATA[
            fileName = this.getClass().getClassLoader().getResource("files/zip-test.zip").getPath();
        ]]>
    </script>
    <var-def name="s3LinkList">
        <s3 bucket="temp_bucket">
            <s3-put path="pdf-test">
                <unzip>
                    <file type="binary" path="${fileName}"/>
                </unzip>
            </s3-put>

         </s3>
     </var-def>

    <var-def name="result">
        <loop item="item" index="i">
            <list>
                <script return="s3LinkList" />
            </list>
            <body>
                <case>
                    <if condition='${item.getWrappedObject().getFilename().endsWith(".txt")}'>
                        <script return="item"/>
                    </if>
                </case>
            </body>
        </loop>
    </var-def>
</config>

---

url-validator

Open plugin description

The plugin is used to detect the availability of the given URL.

The plugin contains the following attributes:

Name	Required	Default	Description
url-column	Yes		The URL of the website to detect the availability.

The plugin returns a list containing instance(s) of the com.freedomoss.crowdcontrol.webharvest.plugin.url.validator.UrlValidationResultDto class.

You can use the following list properties:

• valid
• endUrl
• statusCode
• statusText
• attemptsCount

Example:

<?xml version="1.0" encoding="UTF-8"?>
<config>
  <var-def name="validityList">
    <url-validator url-column="${url}" />
  </var-def>
  <export include-original-data="true">
    <multi-column list="${validityList}">
      <put-to-column-getter name="validity" property="valid"/>
      <put-to-column-getter name="end_url" property="endUrl"/>
      <put-to-column-getter name="status_code" property="statusCode"/>
      <put-to-column-getter name="status_text" property="statusText"/>
      <put-to-column-getter name="attempts_count" property="attemptsCount"/>
    </multi-column>
  </export>
</config>

---

validate

Open plugin description

This is a generic plugin to perform data validation.

The plugin contains the following attributes:

Name	Required	Default	Description
validator	Yes		Fully qualified name of a validator class.
var-def	No		Variable name to store validation results.

You can use the plugin in two ways:

Basic example:

<!-- 1) inside var-def -->
<var-def name="validationResult">
      <validate validator="xxx.xxx.IWebHarvestValidator">String_to_validate</validate>
</var-def>
 
<!-- 2) with var-def as attribute -->
<validate validator="xxx.xxx.IWebHarvestValidator" var-def="validationResult">String_to_validate</validate>

In both cases, the validation results are stored in the provided variable. The validation result object has the following structure:

public class ValidationResult{
    private boolean isValid;
    private List<String> errors = new ArrayList<String>();
    ...
    public boolean isValid();
    public List<String> getErrors();
    ...
}

The validation result provides the following methods:

• isValid() returns a Boolean that tells you whether the string is valid according to the provided validator.
• getErrors() returns a list of strings, provides validation errors if any.

To create a custom validator that can be used in the validation plugin, implement the com.freedomoss.crowdcontrol.webharvest.plugin.validation.IWebHarvestValidator interface.

The currently implemented validators is com.freedomoss.crowdcontrol.webharvest.plugin.validation.iban.IBANValidator that validates IBAN (International Bank Account Number) according to ISO 13616.

---

var-global

Open plugin description

The plugin is used for inserting values of variables defined in the Global Variables Data Store. You can define global variables with values from the Business Process advanced options. If a value is not defined, an exception is thrown into execution.

The plugin contains the following attributes:

Name	Required	Default	Description
name	Yes		Global variable name. Because of storage limitations, it is case-insensitive and treats non-word characters as _ , for example, the Test-key name is treated as test_key.
description	No		Global variable description.

For example, the export value of the test_var global variable in the exported-value column.

Example:

<?xml version="1.0" encoding="UTF-8"?>
<config>
    <export include-original-data="true">
        <single-column name="exported-value">
            <var-global name="test_var"/>
        </single-column>
    </export>
</config>

---

Runtime libraries

The following runtime libraries are available in Bot Configs.

Library	Description
org.apache.commons.lang.StringUtils	Operations on String that are null safe.
com.google.code.gson	Google Gson library to serialize and deserialize Java objects to and from JSON.
commons-httpclient	Feature-rich package implementing the client side of the most recent HTTP standards and recommendations.

troubleshooting

For troubleshooting tips, refer Issues related to execution failures on the Worker side | Plugin execution failures.