Version: 10.3

Standard WebHarvest processors

Below, there is the description of standard WebHarvest processors together with their attributes, syntax, and code samples you can use to design Bot Tasks.

Processor	Description
call	Calls a user-defined function.
case (if/else)	Executes conditional statements and checks sequentially if the conditions specified in the inner `if` elements are satisfied.
config	Root element of every configuration file.
database	Executes a query against a database.
db-param	Specifies the database parameter inside the database element.
empty	Wraps an execution sequence and returns an empty value.
exit	Conditionally breaks the configuration execution.
file	Reads and writes the content or a search directory for specified files.
ftp	Creates an FTP connection and executes valid FTP-based operations against the server.
function	Declares a user-defined function.
html-to-xml	Cleans up the body content and transforms it into valid XML.
json-to-xml	Converts a given JSON content to XML.
loop	Iterates through the specified list and executes the specified body logic for each item.
mail	Sends an email.
mail-attach	Adds an email attachment.
regexp	Searches a body for a given regular expression.
return	Returns a value from the user-defined function.
script	Executes the code written in the specified scripting language.
template	For given text content, the parts enclosed with `${` and `}` are evaluated using the specified scripting engine.
text	Converts an embedded value into a string representation.
tokenize	Splits a given text into elements or tokens.
try-catch	Wraps execution and returns the default value without crashing the whole process.
var	Returns the value of a defined variable.
var-def	Defines new variables or overrides the existing ones with a specified name and value.
while	Loops while the specified condition is satisfied.
xml-to-json	Converts a given XML content to JSON.
xpath	Uses an XPath language expression to search for an XML document.
xslt	Applies XSLT transformation to an XML document.
xquery	Uses an XQuery language expression to query an XML document.
zip	Creates a zip archive by compressing the inner content.

call

Open processor description

The processor calls a user-defined function.

The syntax is as follows:

<call name="function_name">
    <call-param name="function_name">
        body as actual parameter value
    </call-param>
</call>

The attributes are as follows:

Name	Required	Default	Description
`name`	Yes		Name of the user-defined function.

For an example, see the example of the function processor.

case

Open processor description

The processor executes conditional statements and checks sequentially if the conditions specified in the inner if elements are satisfied. If it finds one that is satisfied, it returns its body as the result. If no true statement is found, the execution result is the else statement body, if specified. Otherwise, it is an empty value.

The syntax is as follows:

<case>
    [<if condition="expression"> if body </if>]
    [<else> else body </else>]
</case>

The attributes are as follows:

Name	Required	Default	Description
`condition`	Yes		If true (`yes`), the `if` body is evaluated.

Example:

<var-def name="contact">
    <xpath expression="//a[contains(., 'contact')]/@href">
        <var name="pageContent"/>
    </xpath>
</var-def>
<var-def name="contactMail">
    <case>
        <if condition="${contact.toString() != ''}">
            <var name="contact"/>
        </if>
        <else>
            Contact is not defined!
        </else>
    </case>
</var-def>

The conditional processor checks if the previous XPpath search found any contact information on the page.

config

Open processor description

The processor is the root element of every configuration file.

The syntax is as follows:

<config charset="charset_value" scriptlang="default_script_lang">
    configuration body
</config>

The attributes are as follows:

Name	Required	Default	Description
`charset`	No	UTF-8	Defines the default charset to be used throughout the configuration. Every processor that needs charset information utilizes this value unless any other is set explicitly.
`scriptlang`	No	beanshell	Defines the default scripting engine to be used throughout the configuration. Allowed values are `groovy` or `beanshell`. The default engine is used wherever no other is specified. The script and template processors enable specifying several scripting engines within the same WebHarvest configuration. In this way, you can mix several different scripting languages.

Example:

<config charset="ISO-8859-1" scriptlang="groovy">
    <file action="write" path="squares.txt">
        <script return="${[1, 2, 3, 4, 5].collect(square)}"><![CDATA[
            square = { it * it }
        ]]></script>
    </file>
</config>

The file processor uses the explicitly defined ISO-8859-1 encoding, whereas the script processor uses the explicitly defined Groovy language.

database

Open processor description

The processor executes a query against a database.

If used programmatically, the JDBC driver library file(s) should be on the classpath. If used standalone, they should be on the same path with the WebHarvest executable.

With the SELECT sql statement, the processor returns a list of row objects. They can be accessed with special accessor methods:

getColumnCount() returns the number of columns returned.
getColumnName(index) returns a name for the column number.
get(column_index) returns a field value for the column number.
get(column_name) returns a field value for the column name.

The entire list of returned DB rows can be accessed by an index to get an individual row:

<mydbvar>.get(rowindex)

For example:

mydb.get(0).get("image")

The syntax is as follows:

<database connection="jdbc connection string"
          jdbcclass="full named jdbc class"
          username="username"
          password="password"
          autocommit="autocommit"
          max="max rows returned">
    select, insert or delete SQL query
</database>

The attributes are as follows:

Name	Required	Default	Description
`connection`	Yes		Properly formatted JDBC string for the database. Depends on the database or driver vendor.
`jdbcclass`	Yes		Fully qualified class name of the JDBC driver.
`username`	No		Username to access the database.
`password`	No		Password to access the database.
`autocommit`	No	True	Defines whether the commit is performed automatically after executing the query.
`max`	No	No limit	Maximum number of returned rows from the `SELECT` statement.

Example 1:

<var-def name="employees">
    <database connection="jdbc:microsoft:Sqlserver://myserver:1433;databaseName=mycompany;user=sa;password=hehehe"
              jdbcclass="com.microsoft.jdbc.sqlserver.SQLServerDriver">
        select name, salary from employee
    </database>
</var-def>

<loop item="emp">
    <list>
        <var name="employees"/>
    </list>
    <body>
        <template>Salary of ${emp.get("name")} is ${emp.get("salary")}</template>
    </body>
</loop>

Example 2:

<database connection="jdbc:microsoft:Sqlserver://myserver:1433;databaseName=mycompany;user=sa;password=hehehe"
          jdbcclass="com.microsoft.jdbc.sqlserver.SQLServerDriver">
    <template>
        insert into news (id, url, text, source)
        values (${myId}, '${myUrl}', '${myText}', '${mySource}')
    </template>
</database>

db-param

Open processor description

The processor specifies the database parameter inside the database element. Can be used for storing BLOBs (Binary Large Objects).

The syntax is as follows:

<db-param type="param_type">
    parameter value
</db-param>

The attributes are as follows:

Name	Required	Default	Description
`type`	No	`binary` if its value is recognized as binary, `text` otherwise.	Type of the parameter. Valid values are as follows: `int`, `long`, `double`, `text`, and `binary`.

Example:

<database connection="jdbc:mysql://myserver/mydb"
          jdbcclass="com.mysql.jdbc.Driver"
          username="myuser"
          password="mypass">
    insert into logos (id, img)
    values ( 1, <db-param><http url='${myImageUrl}'/></db-param> )
</database>

empty

Open processor description

The processor wraps an execution sequence and returns an empty value. This element is used when the execution result is not essential.

The syntax is as follows:

<var-def name="amazonContent">
    <empty>
        <http url="http://www.amazon.com" />
    </empty>
</var-def>

Example:

<file action="write" path="test/amazon_home.html">
    <empty>
        <var-def name="amazonContent">
            <http url="http://www.amazon.com"/>
        </var-def>
    </empty>
    <template>
        <![CDATA[
            <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
        ]]>
        ${amazonContent}
    </template>
</file>

A new variable is created, but its value is not included in the result because it is inside an empty element. Instead, its value is used in the subsequent template processor.

exit

Open processor description

The processor conditionally breaks the configuration execution.

The syntax is as follows:

<exit condition="condition" message="message" />

The attributes are as follows:

Name	Required	Default	Description
`condition`	No	True	Condition that determines if an execution is to be stopped. Must be a Boolean value: `true`, `yes`, `false`, `no`.
`message`	No		Optional message to a user if the configuration is stopped. Included in the logging information, or a dialog pops up if WebHarvest is used in the GUI mode.

Example:

<exit condition='${!sys.isVariableDefined("username")}' message="No username provided!" />

The configuration execution should be stopped if the username variable is not defined.

file

Open processor description

The processor reads and writes the content or a search directory for specified files.

The syntax is as follows:

<file action="file_action" path="file_path" type="file_type" charset="charset_of_text_file" listdirs="listdirs" listfiles="listfiles" listrecursive="listrecursive" listfilter="listfilter">
    body defining content of the file if action="write" or action="append" '
</file>

The attributes are as follows:

Name	Required	Default	Description
`action`	No	Read	File action. Valid values are `read`, `append`, `write`, and `list`.
`path`	Yes		File path, relative to the working directory.
`type`	No	Text	File type: `text` or `binary`.
`charset`	No	Default charset for config.	Charset for text files. Has no effect if `type` is `binary`.
`listdirs`	No	Yes	Indicates whether to list directories: `action="list"`.
`listfiles`	No	Yes	Indicates whether to list files: `action="list"`.
`listrecursive`	No	No	Indicates whether to search directories recursively: `action="list"`.
`listfilter`	No		Filename pattern to search for. For example, `*` is the substitute for any sequence, while `?` stands for any character. Works only for `action = "list"`.

Example 1:

<file action="write" path="123.txt">
    <file action="read" path="1.txt"/>
    -----------------------------------
    <file action="read" path="2.txt"/>
    -----------------------------------
    <file action="read" path="3.txt"/>
</file>

A new file containing appended contents of three existing files separated by lines is created.

Example 2:

<file action="write" path="c:/images/alljpegs.zip" type="binary">
    <zip>
        <loop item="filename">
            <list>
                <file path="c:/images/" action="list" listfilter="*.jpg" />
            </list>
            <body>
                <zip-entry name="${sys.getFilename(filename.toString())}">
                    <file type="binary" path="${filename}"/>
                </zip-entry>
            </body>
        </loop>
    </zip>
</file>

A zip file is created. The file comprises all JPEG images taken from the specified directory.

ftp

Open processor description

The processor creates an FTP connection and executes valid FTP-based operations against the server:

ftp-list
ftp-get
ftp-put
ftp-del
ftp-mkdir
ftp-rmdir

The syntax is as follows:

<ftp server="server" port="port" username="username" password="password" account="account" remotedir="remotedir">
    [<ftp-list path="path" listfiles="listfiles" listdirs="listdirs" listlinks="listlinks" listfilter="listfilter"/>]* [<ftp-get path="path"/>]
    [<ftp-put path="path" charset="charset"> content to save </ftp-put>]
    [<ftp-del path="path"/>]
    [<ftp-mkdir path="path"/>]
    [<ftp-rmdir path="path"/>]
</ftp>

The attributes are as follows:

Name	Required	Default	Description
`server`	Yes		FTP server address.
`port`	No	21	FTP server port.
`username`	Yes		FTP server username.
`password`	Yes		FTP server password.
`account`	No		FTP server account name.
`remotedir`	No		Working remote directory on the FTP server.
`path`	Yes		Path of the file or directory to be accessed, added, or removed.
`listfiles`	No	Yes	Defines whether to include files in the list.
`listdirs`	No	Yes	Defines whether to include directories in the list.
`listlinks`	No	Yes	Defines whether to include links in the list.
`listfilter`	No		Filter used for listing files. Can include `` and `?`, for example, `my.ex?`.

function

Open processor description

The processor declares a user-defined function.

The syntax is as follows:

<function name="function_name">
    function body
</function>

The attributes are as follows:

Name	Required	Default	Description
`name`	Yes		Name of the user-defined function.

Example:

<function name="download-multipage-list">
    <return>
        <while condition="${pageUrl.toString().trim() != ''}" maxloops="${maxloops}" index="i">
            <empty>
                <var-def name="content">
                    <html-to-xml>
                        <http url="${pageUrl}"/>
                    </html-to-xml>
                </var-def>
                <var-def name="nextLinkUrl">
                    <xpath expression="${nextXpath}">
                        <var name="content"/>
                    </xpath>
                </var-def>
                <var-def name="pageUrl">
                    <template>${sys.fullUrl(pageUrl, nextLinkUrl)}</template>
                </var-def>
            </empty>
            <xpath expression="${itemXPath}">
                <var name="content"/>
            </xpath>
        </while>
    </return>
</function>
 
<var-def name="imgLinks">
    <call name="download-multipage-list">
        <call-param name="pageUrl">http://images.google.com/images?q=harvest&hl=en&btnG=Search+Images&nojs=1</call-param>
        <call-param name="nextXPath">//a[@shape='rect' and .='Next']/@href</call-param>
        <call-param name="itemXPath">//img[contains(@src, 'images?q=tbn')]/@src</call-param>
        <call-param name="maxloops">5</call-param>
    </call>
</var-def>

The download-multipage-list function is defined for multiple extractions. It collects URLs from a series of pages, where the XPath expression parameter is used to determine the URL of the next page with links if there is one. The situation is typical for a list of products or search results spanning multiple web pages.

After that, the function is called with specified parameters to collect image links from the Google images search, limiting the number of resulting pages to five.

html-to-xml

Open processor description

The processor cleans up the body content and transforms it into valid XML. Typically, the body is HTML resulting from the execution of the http processor. The actual job of parsing and cleaning is delegated to the HtmlCleaner tool. Although no special tuning is needed in most cases, you can configure a cleaner by defining several parameters using the processor's attributes.

The syntax is as follows:

<html-to-xml outputtype="..." advancedxmlescape="..." usecdata="..." specialentities="..." unicodechars="..." omitunknowntags="..." treatunknowntagsascontent="..." omitdeprtags="..." treatdeprtagsascontent="..." omitcomments="..." omithtmlenvelope="..." allowmultiwordattributes="..." allowhtmlinsideattributes="..." namespacesaware="..." prunetags="..." omitxmldecl="...">
    body as html to be cleaned
</html-to-xml>

The attributes are as follows:

Name	Required	Default	Description
`outputtype`	No	Simple	Defines how the resulting XML is to be serialized. Allowed values are `simple`, `compact`, `browser-compact`, and `pretty`.
`advancedxmlescape`	No	True	If this parameter is set to `true`, the ampersand sign (&) that precedes valid XML character sequences (&XXX;) is not escaped with &XXX;
`usecdata`	No	True	If `true`, `HtmlCleaner` treats the `SCRIPT` and `STYLE` tag content as CDATA sections. Otherwise, it is regarded as ordinary text (special characters are escaped).
`specialentities`	No	True	If `true`, special HTML entities (ô, ‰, ×) are replaced with Unicode characters they represent (ô, ‰, ×). This doesn't include ×, <, >, ", '.
unicodechars	No	True	If `true`, HTML characters represented by their codes in the &#XXXX; format are replaced with real Unicode characters.
`omitunknowntags`	No	False	Defines whether to skip (ignore) unknown tags during cleanup.
`treatunknowntagsascontent`	No	False	Defines whether to treat unknown tags as ordinary content. That means `<something...>` is transformed into `<something...>`. This attribute is applicable only if `ifomitUnknownTags` is set to `false`.
`omitdeprtags`	No	False	Defines whether to skip (ignore) deprecated `HTML` tags during cleanup.
`treatdeprtagsascontent`	No	False	Defines whether to treat deprecated tags as ordinary content. That means `<font...>` is transformed into `<font...>`. The attribute is applicable only if `omitDeprecatedTags` is set to `false`.
`omitcomments`	No	False	Defines whether to skip HTML comments.
`omithtmlenvelope`	No	False	Defines whether to remove `HTML` and `BODY` tags from the resulting XML and use the first tag in the `BODY` section instead. If the `BODY` section doesn't contain any tags, the attribute has no effect.
`allowmultiwordattributes`	No	True	Tells the parser whether to allow attribute values consisting of multiple words or not. If true, the `att="a b c"` attribute stays as it is. If `false`, the parser splits it into `att="a" b="b" c="c"`, which is the default browser behavior.
`allowhtmlinsideattributes`	No	False	Tells the parser whether to allow HTML tags inside attribute values. For example, when the flag is set, `att="here is <a href='xxxx'>link</a>"` remains as it is. If not, the parser ends the attribute value after `here is` . The flag makes sense only if `allowMultiWordAttributes` is set as well.
`namespacesaware`	No	True	If `true`, any namespace prefixes found during parsing are preserved. All needed XML namespace declarations are added to the root element. If `false`, all namespace prefixes and all XMLNS namespace declarations are stripped.
`prunetags`	No	Empty string	Comma-separated list of tags to be removed completely (with all nested elements) from the XML tree after parsing. For example, if `pruneTags` is `"script,style"`, the resulting XML contains scripts and styles.
`trimattributevalues`	No	True	By default, the white spaces are trimmed from the start and end of attribute values. This can be disabled to get untouched (original) attribute values in the resulting XML.
`omitdoctype`	No	False	Defines whether to remove the `HTML DOCTYPE` section from the resulting XML.
`omitxmldecl`	No	False	Defines whether to put the XML declaration line at the beginning of the resulting XML or skip it.
`useemptyelementtags`	No	True	Specifies how to serialize tags with an empty body. If `true`, a compact notation is used: `<xxx/>`. Otherwise, it is `<xxx></xxx>`.
`hyphenreplacement`	No	=	XML doesn't allow the double hyphen sequence (--) inside comments. This attribute defines the substitute for the double hyphen when it is encountered during parsing.
`booleanatts`	No	Self	Tells the cleaner what value to give to Boolean attributes, such as `checked`, `selected`, and so on. Allowed values are as follows: `self`: the attribute value is the same as the attribute name: `checked = "checked"`. `empty`: the attribute value is an empty string: `checked = ""`. `true`: the attribute value is `true`: `checked = "true"`.

Example:

<html-to-xml outputtype="pretty">
    <http url="http://www.motors.ebay.com"/>
</html-to-xml>

The script downloads the www.motors.ebay.com page and cleans it up, producing pretty-printed XML content.

json-to-xml

Open processor description

The processor converts a given JSON content to XML. See also JSON manipulation plugins.

The syntax is as follows:

<json-to-xml>
    JSON content
</json-to-xml>

loop

Open processor description

The processor iterates through the specified list and executes the specified body logic for each item. The result is a list of processed bodies.

caution

Incorrect usage of the plugin can lead to a Worker crash. To avoid such issues, refer to Optimize memory usage when designing Bot Tasks.

The syntax is as follows:

<loop item="item_var_name" index="index_var_name" maxloops="max_loops" filter="list_filter" empty="true">
    <list>
        body as list value
    </list>
    <body>
        body for each list item
    </body>
</loop>

The attributes are as follows:

Name	Required	Default	Description
`item`	No		Name of the variable that takes the value of the current list item.
`index`	No		Name of the index variable. The initial value for the first loop is `1`.
`maxloops`	No		Limits the number of iterations. There is no limit if not specified.
`filter`	No		Expression for filtering the iteration list. It consists of an arbitrary number of restrictions separated by commas. There are the following types of restrictions: `[n]-[m]` specifies the index range, for example, `3-6`, `-5`. `[n][:][m]` specifies a sublist starting at the `n` index and including items at indexes `n+m`, `n+2m`, `n+3m`, ..., for example, `1:2` for all odd, `2:2` for all even. `unique` removes duplicates from the list, comparing the string values of the list items. A valid filter is a combination of allowed restrictions, for example, `1-20`, `1:2`, `unique`.
`empty`	No	`no`	If the value is `true`, it equals the surrounding body by the `empty` element, producing an empty iteration result.

Example:

<loop item="link" index="i" filter="unique">
    <list>
        <xpath expression="//img/@src">
            <html-to-xml>
                <http url="http://www.yahoo.com"/>
            </html-to-xml>
        </xpath>
    </list>
    <body>
        <file action="write" type="binary" path="images/${i}.gif">
            <http url="${sys.fullUrl('http://www.yahoo.com', link)}"/>
        </file>
    </body>
</loop>

loop iterates over all unique image URLs from the Yahoo website. For each URL, it downloads an image and saves it to the file system.

Iterating over an input data array:

<config>
    <!--- result_list - is the JSON formatted array of values from the previous step -->
    <script  return="xmlresult"><![CDATA[
    import org.json.XML;
    import java.util.List;
    import java.util.ArrayList;
    import org.json.JSONObject;
    import org.json.JSONArray;
    import com.freedomoss.workfusion.utils.gson.GsonUtils;

    JSONArray arr = new JSONArray(result_list.toString());
    JSONObject obj = new JSONObject();
    obj.append("results",arr);
    xmlresult = XML.toString(obj,"uploadrequest");
    log.error(xmlresult);
    sys.defineVariable("xml_result",xmlresult,true);

    List result = new ArrayList();
    ]]>
    </script>

    <var-def name="requests">
        <xpath expression="//array">
                <var name="xml_result"/>
        </xpath>
    </var-def>

    <loop empty="true" item="request">
        <list><var name="requests"/></list>
        <body>
            <!--read values from each element of the list-->
            <var-def name="doc_title">
                <xpath expression="//document_title/text()">
                    <var name="request"/>
                </xpath>
            </var-def>
            <!--- execute your logic for a single element from the list -->
            <script><![CDATA[
                log.error(title: doc_title.toString());
                            result.add("doc_title.toString()");
            ]]></script>
        </body>
    </loop>
    <export include-original-data="true">
    <single-column name="loop_results" value="${GsonUtils.GSON.toJson(result)}"></single-column>
</export>
</config>

The loop_result output value contains the results of your step executed for each array element from the input:

[
    {
        "document_title":"Proof of document",
        "content_group_f":2,
        "document_link_f":"http://unknow.name/1474360372873_testid.pdf"
    },
    {
        "document_title":"Proof of identity",
        "content_group_f":1,
        "document_link_f":"http://unknow.name/1474360372888_testid.pdf"
    },
    {
        "document_title":"Proof 2",
        "content_group_f":10,
        "document_link_f":"http://unknow.name/1474360372880_testid.pdf"
    },
    {
        "document_title":"Proof of income",
        "content_group_f":10,
        "document_link_f":"http://unknow.name/1474360372884_testid.pdf"
    }
]

mail

Open processor description

The processor sends an email.

The syntax is as follows:

<mail smtp-host="smtp server"
    smtp-port="smtp server port"
    type="content type"
    from="sender"
    reply-to="reply-to header"
    to="to"
    cc="cc"
    bcc="bcc"
    subject="subject"
    charset="charset"
    username="smtp username"
    password="smtp password"
    security="smtp security type">

    mail content with optional attachments (mail-attach elements)

</mail> 

The attributes are as follows:

Name	Required	Default	Description
`smtp-host`	Yes		SMTP server host.
`smtp-port`	No	25	SMTP server port.
`type`	No	Text	Content type of the mail body—text or HTML.
`from`	Yes		Sender's email address.
`reply-to`	No		Email address where replies are to be sent.
`to`	Yes		Comma-separated list of recipient email addresses.
`cc`	No		Comma-separated list of cc email addresses.
`subject`	No		Subject of the email.
`charset`	No	Default charset for the configuration.	Charset of the email.
`username`	No		SMTP server username.
`password`	No		SMTP server password.
`security`	No	None	SMTP server security type: `none`, `ssl`, or `tls`.

Example:

<?xml version="1.0" encoding="UTF-8"?>

<config>
    <var-def name="acronym">
        SPA
    </var-def>
    <mail smtp-host="smtp.yandex.com"
          smtp-port="587"
          type="html"
          from="username@yandex.com"
          to="username@yandex.com"
          subject="Some test subject"
          charset="UTF-8"
          username="username@yandex.com"
          password="super_secure_password"
          security="ssl">

        Using HTML content in your email:
        <![CDATA[
            <p><a href="https://workfusion.com">WorkFusion Inc.</a> is providing <strong>SPA*</strong></p>
        ]]>

        <![CDATA[<hr><em>]]>
        *<var name="acronym"/> - Smart Process Automation
        <![CDATA[</em>]]>

    </mail>
</config>

mail-attach

Open processor description

The processor adds an email attachment. Can be used only as part of the mail processor of the html type.

The syntax is as follows:

<mail-attach name="name" mimetype="mimetype" inline="inline">
    body of the attachment
</mail-attach>

The attributes are as follows:

Name	Required	Default	Description
`name`	No	Attachment N.	Name of the attachment.
`mimetype`	No	Image or JPEG for inline attachments, application/octet-stream otherwise.	Mime type of the attachment.
`inline`	No	No	Defines whether the attachment is embedded into the mail body.

Example:

<mail from="my@my.com" smtp-host="smtp.gmail.com" to="myaccount@gmail.com" type="html" username="myusername" password="mypassword" security="tsl" subject='Photos from the ...'>
    Here is me with ... <![CDATA[ <img src="]]>
    <mail-attach inline="true">
        <file path="myphoto1.jpg" type="binary"/>
    </mail-attach>
    <![CDATA[ "> ]]> And this is ... <![CDATA[ <img src="]]>
    <mail-attach inline="true">
        <file path="myphoto2.jpg" type="binary"/>
    </mail-attach>
    <![CDATA[ "> ]]>
</mail>

regexp

Open processor description

The processor searches a body for a given regular expression. Optionally, it replaces found occurrences with a specified pattern. If the body is a list of values, regexp is applied to every item. The final execution result is a list.

The syntax is as follows:

<regexp replace="true_or_false" max="max_found_occurrences" flag-canoneq="flag-canoneq" flag-caseinsensitive="flag-caseinsensitive" flag-dotall="flag-dotall" flag-multiline="flag-multiline" flag-unicodecase="flag-unicodecase">
    <regexp-pattern> body as pattern value </regexp-pattern>
    <regexp-source> body as the text source </regexp-source>
    [<regexp-result> body as the result </regexp-result>]
</regexp>

For each group inside the search pattern and each found occurrence, variables with the _<group_number> name are created. For details about groups, search for Regular Expression tutorials on the web.

The attributes are as follows:

Name	Required	Default	Description
`replace`	No	False	Logical value defining whether found occurrences of a regular expression are to be substituted. Valid values are: `true` or `false`, or `yes` or `no`. If the value is `true` (`yes`), `regexp-result` has to be specified with a substitute value.
`max`	No		Limits the number of found pattern occurrences. There is no limit if not specified.
`flag-canoneq`	No	No	Enables canonical equivalence.
`flag-caseinsensitive`	No	No	Enables case-insensitive matching.
`flag-dotall`	No	Yes	Enables the dotall mode.
`flag-multiline`	No	No	Enables the multiline mode.
`flag-unicodecase`	No	Yes	Enables Unicode-aware case folding.

Example 1:

<regexp>
    <regexp-pattern>([_\w\d]*)[\s]*=[\s]*([\w\d\s]*+)[\,\.\;]*</regexp-pattern>
    <regexp-source> var1= test1, var2 = bla bla; index=16; city = Delhi,town=Kingston; </regexp-source>
    <regexp-result>
        <template>Value of variable "${_1}" is "${_2}"!</template>
    </regexp-result>
</regexp>

The regular expression is looking for the specified pattern in two strings, producing a list of five values: Value of variable "var1" is "test1"!, Value of variable "var2" is "bla bla"!, and so on.

Example 2:

<regexp replace="true">
    <regexp-pattern>[\s]*[\,\.\;][\s]*</regexp-pattern>
    <regexp-source>
        var1= test1, var2 = bla bla; index=16; city = Delhi,town=Kingston;
    </regexp-source>
    <regexp-result>
        <template>|</template>
    </regexp-result>
</regexp>

return

Open processor description

The processor returns a value from the user-defined function.

The syntax is as follows:

<return> body as return value </return>

For an example, see the example of the function processor.

script

Open processor description

The processor executes the code written in the specified scripting language—Groovy or BeanShell.

caution

Eclipse supports debugging for Groovy scripts only. BeanShell is not recommended for creating new scripts as it is deprecated (backward compatibility only).

The body of the script processor is executed in the specified language. Optionally, the processor can return the evaluated expression set for the return attribute.

All variables defined during the configuration execution are also available in the script processor. However, variables used throughout WebHarvest are not simple. They are org.webharvest.runtime.variables.Variable objects (internal WebHarvest class) that expose convenient methods:

String toString()
byte[] toBinary()
boolean toBoolean()
int toInt()
long toLong()
double toDouble()
double toDouble()
Object[] toArray()
java.util.List toList()
Object getWrappedObject()

To push a value back to WebHarvest after the script is executed, use the sys.defineVariable(varName, varValue, [overwrite]) command. It creates a wrapper around the specified value: list variables for java.util.List, arrays, and simple variables for other objects. For an illustration, see the example below.

Once created, each script engine used in the single WebHarvest configuration preserves its variable context throughout the configuration, meaning that all variables and objects are available in further script processors using the same language.

The syntax is as follows:

<script language="script_language" return="value_to_return">
    body as script
</script>

The attributes are as follows:

Name	Required	Default	Description
`language`	No	Default scripting language if specified in the `config` element; otherwise, `beanshell`.	Defines which scripting engine is used in the processor. Valid values are `groovy` or `beanshell`.
`return`	No	Empty value.	Specifies what the processor is to evaluate in the end and return as the processing value.

Example:

<config>
    <var-def name="birthDate"> 11/4/1958 </var-def>
    <var-def name="web_harvest_day_variable">
        <script return="namedDay.toUpperCase()"><![CDATA[
            tokenizer = new StringTokenizer(birthDate.toString(), "./-\\");
            day = Integer.parseInt(tokenizer.nextToken());
            month = Integer.parseInt(tokenizer.nextToken());
            year = Integer.parseInt(tokenizer.nextToken());
            Calendar cal = Calendar.getInstance();
            cal.set(Calendar.DAY_OF_MONTH, day);
            cal.set(Calendar.MONTH, month-1);
            cal.set(Calendar.YEAR, year);
            switch( cal.get(Calendar.DAY_OF_WEEK) ) {
                case 0 :
                    namedDay = "Sunday";
                    break;
                case 1 :
                    namedDay = "Monday";
                    break;
                case 2 :
                    namedDay = "Tuesday";
                    break;
                case 3 :
                    namedDay = "Wednesday";
                    break;
                case 4 :
                    namedDay = "Thursday";
                    break;
                case 5 :
                    namedDay = "Friday";
                    break;
                default:
                    namedDay = "Saturday";
                    break;
            }
        ]]></script>
    </var-def>
    <template> The day when you were born was ${namedDay}. </template>
    <file action="write" path="day.txt">
        <var name="web_harvest_day_variable"/>
    </file>
</config>

Once defined, the script's internal variables are available in both script and template processors (namedDay).

template

Open processor description

For given text content, the parts enclosed with ${ and } are evaluated using the specified scripting engine. If no scripting language is specified, the default one is used; see the config element.

The syntax is as follows:

<template language="script_language">
    body as text for templating
</template>

The attributes are as follows:

Name	Required	Default	Description
`language`	No	Default config language.	Specifies the script language to be used for evaluation of the parts enclosed with `${` and `}`. Valid values are `groovy` or `beanshell`.

Example:

<var-def name="content">
    <file path="textdata/products.txt"/>
</var-def>
<var-def name="changedContent">
    <template>
        ${sys.datetime("yyyy-MM-dd, HH:mm:ss")} ${sys.lf} ---------------------------------------------------- ${sys.lf} ${my.process(content.toString())}
    </template>
</var-def>

The template uses built-in constants, functions, and user-defined objects from the variable context to produce the desired content.

text

Open processor description

The processor converts an embedded value into a string representation.

The syntax is as follows:

<text charset="charset" delimiter="delimiter">
    wrapped body
</text>

The attributes are as follows:

Name	Required	Default	Description
`charset`	No	Default configuration's charset.	Charset used if a body is converted from binary into a text value.
`delimiter`	No	Newline character.	Delimiter string used to separate items concatenated into a single string.

Example:

<var-def name="digits">
    <while condition="${i.toInt() != 10}" index="i">
        <template>${i}</template>
    </while>
</var-def>
<file action="write" path="/test/replaced23.txt">
    <regexp replace="true">
        <regexp-pattern>(.*)(2.*3)(.*)</regexp-pattern>
        <regexp-source>
            <text><var name="digits"/></text>
        </regexp-source>
        <regexp-result>
            <template>${_1}here were 2 and 3${_3}</template>
        </regexp-result>
    </regexp>
</file>

The digits variable is defined using the while processor, producing a sequence of nine values. Next, the regexp processor is invoked to search-replace the variable's value. The text processor is used to concatenate all digit values into a single one.

Without the text processor, a regular expression search is applied to each item (every digit) on the list. This means no replacement because there is no 2*3 sequence.

tokenize

Open processor description

The processor splits a given text into elements (tokens).

The syntax is as follows:

<tokenize delimiters="delimiters"
          trimtokens="trimtokens"
          allowemptytokens="allowemptytokens">
    content to tokenize
</tokenize>

The attributes are as follows:

Name	Required	Default	Description
`delimiters`	No	Newline character.	Defines characters to be used as token delimiters.
`trimtokens`	No	Yes	Indicates whether to trim resulting tokens.
`allowemptytokens`	No	No	Indicates whether to include empty tokens into the resulting list consisting only of whitespaces.

try-catch

Open processor description

The processor wraps execution and, for any recoverable exceptions, returns the default value without crashing the whole process.

The syntax is as follows:

<try>
    <body> try body </body>
    <catch> catch body </catch>
</try>

Example:

<var-def name="reportText">
    <try>     
        <body>
            <file path="data/report.txt"/>
        </body>
        <catch>
            No report file!
        </catch>
    </try>
</var-def>

A file read exception is caught if occurred. The default value is stored in a variable. To get or log an error or error description in the <catch> block, use the _exception, _exception_message, and _exception_stacktrace objects as shown below:

<config>
    <try>
        <body>
            <script><![CDATA[ throw new RuntimeException("Something bad happenned!"); ]]></script>
        </body>
        <catch>
            <script><![CDATA[
                log.info(_exception_message.getWrappedObject(), _exception.getWrappedObject());
                log.info(_exception_stacktrace.getWrappedObject());
            ]]></script>
        </catch>
    </try>
</config>

var

Open processor description

The processor returns the value of a defined variable. It throws an exception if the variable is not defined.

The syntax is as follows:

<var name="variable_name"/>

The attributes are as follows:

Name	Required	Default	Description
`name`	Yes		Variable name.

Example:

<var-def name="searchEngine">google</var-def>
<var-def name="${searchEngine}Content"><http url="http://www.${searchEngine}.com"/></var-def>
<file action="write" path="data/${searchEngine}_content.html">
    <var name="${searchEngine}Content"/>
</file>

After execution, the google_content.html file comprises the content of the www.google.com page.

var-def

Open processor description

The processor defines new variables or overrides the existing ones with a specified name and value.

The syntax is as follows:

<var-def name="variable_name" overwrite="overwrite_existing">
    body as a value of the variable
</var-def>

The attributes are as follows:

Name	Required	Default	Description
`name`	Yes		Variable name. Must be valid like in most programming languages.
`overwrite`	No	True	Boolean value (`true` or `false`) indicating if the existing variable with the same name should be overwritten or not.

Example:

<var-def name="digitList">
    <while condition="true" index="i" maxloops="9">
        <var-def name="digit${i}"><template>${i}</template></var-def>
    </while>
</var-def>

The digitList variable is defined, which is a sequence of nine values (digits from one to nine) and ten simple variables (digit1, digit2, ..., digit9) with values ranging from one to nine.

while

Open processor description

The processor loops while the specified condition is satisfied. The result is a list of processed bodies obtained in each iteration.

caution

Incorrect usage of the plugin can lead to a Worker crash. To avoid such issues, refer to Optimize memory usage when designing Bot Tasks.

The syntax is as follows:

    <script><![CDATA[
        num = 7;    
        exportItems = new ArrayList();
    ]]></script>

    <while condition="true" maxloops="${num}" index="ind" empty="true">
        <script><![CDATA[
            Map exportItem = new HashMap();
            exportItem.put("first_name", "Name" + ind);
            exportItem.put("last_name", "Surname" + ind);
            exportItem.put("employee_id", "100" + ind);
            exportItems.add(exportItem);
        ]]></script>
    </while>

The attributes are as follows:

Name	Required	Default	Description
`condition`	Yes		Expression evaluated for every loop. If its value is `true`, the body is executed.
`index`	No		Name of the index variable. The initial value for the first loop is `1`.
`maxloops`	No		Limits the number of iterations. There is no limit if not specified.
`empty`	No	No	If set to `true`, it equals the surrounding body by the `empty` element, producing an empty iteration result.

xml-to-json

Open processor description

The processor converts a given XML content to JSON.

The syntax is as follows:

<xml-to-json>
    XML content
</xml-to-json>

Example:

 <var-def name="outputLink">
   <s3 bucket="vr1677">
       <s3-put path="jdoe/my_super_file.csv" acl="PublicRead" content-type="text/csv; charset=utf-8" content-disposition="inline">
           <list-to-csv>
               <json expression="$.row">
                   <xml-to-json>
                       <datastore name="ds_test">
                           select * from @this;
                       </datastore>
                   </xml-to-json>
               </json>
           </list-to-csv>
       </s3-put>
   </s3>
</var-def>

xpath

Open processor description

The processor uses an XPath language expression to search for an XML document.

The syntax is as follows:

<xpath expression="xpath_expression">
    body as xml
</xpath>

The attributes are as follows:

Name	Required	Default	Description
`expression`	Yes		XPath language expression.

Example:

<xpath expression="//a/@href">
    <html-to-xml>
        <http url="http://www.nba.com/"/>
    </html-to-xml>
</xpath>

The result is a sequence of links from the page retrieved from www.nba.com.

xslt

Open processor description

The processor applies XSLT transformation to an XML document.

The syntax is as follows:

<xslt>
    <xml>
        body as xml
    </xml>
    <stylesheet>
        body as xsl
    </stylesheet>
</xslt>

Example:

<xslt>
    <xml>
        <html-to-xml>
            <http url="${url}"/>
        </html-to-xml>
    </xml>
    <stylesheet>
        <file path="stylesheets/tree.xsl"/>
    </stylesheet>
</xslt>

An XSLT transformation from the file is applied to the downloaded content.

xquery

Open processor description

The processor uses an XQuery language expression to query an XML document.

The syntax is as follows:

<xquery>
    [<xq-param name="xquery_param_name" [type="xquery_param_type"]>
        body as xquery parameter value
    </xq-param>] *
    <xq-expression>
        body as xquery language construct
    </xq-expression>
</xquery>

The attributes are as follows:

Name	Required	Default	Description
name	Yes		Name of the XQuery parameter.
type	No	`node()`	XQuery parameter type, which is one of the values: `node()`, `integer`, `long`, `float`, `double`, `boolean`, `string`, `node()`, `integer`, `long`, `float`, `double`, `boolean`, `string*`.

Optionally, it is allowed to specify multiple external parameters for the query. In most cases, at least one containing XML document is needed. For every specified xquery parameter, the declaration inside the xq-expression has the following format:

declare variable $<xquery_param_name> as <xquery_param_type> external;

The declaration is required to match the name and the type of the processed parameter. Valid parameter types supported by WebHarvest are as follows:

node()
integer
long
float
double
boolean
string

Analog sequence types include:

node()*
integer*
long*
float*
double*
boolean*
string*

If not specified, the default XQuery parameter is node().

Example:

<xquery>
    <xq-param name="doc">
        <html-to-xml>
            <http url="${sys.fullUrl(startUrl, articleUrl)}"/>
        </html-to-xml>
    </xq-param>
    <xq-expression><![CDATA[
        declare variable $doc as node() external;

        let $author := data($doc//div[@class="byline"])
        let $title := data($doc//h1)
        let $text := data($doc//div[@id="articleBody"])
            return
                <article>
                    <title>{$title}</title>
                    <author>{$author}</author>
                    <text>{$text}</text>
                </article>
    ]]></xq-expression>
</xquery>

xquery is applied to the downloaded page. The result is an XML containing information about newspaper articles.

zip

Open processor description

The processor creates a zip archive by compressing the inner content defined by the zip-entry elements. To unzip, use the unzip plugin.

The syntax is as follows:

<zip>
    ...
    [<zip-entry name="name" charset="charset">
        entry content
    </zip-entry>]*
    ...
</zip>

The attributes are as follows:

Name	Required	Default	Description
`name`	Yes		Name of the file inside the zip archive.
`charset`	No	Default charset for the configuration.	Charset of the text file inside the zip archive.

Example:

<zip>
    <loop item="filename" index="i">
        <list>
            <var name="myfilenames"/>
        </list>
        <body>
            <zip-entry name="file${i}.xls">
                <file path="${filename}" type="binary"/>
            </zip-entry>
        </body>
    </loop>
</zip>

An archive is created that includes a list of specified files. The zip archive can further be sent via email, saved to a database or a file system so that the zip element can be inside mail, database, file, or any other valid processor.

Optimize memory usage when designing Bot Tasks

When designing a Bot Task, avoid using the while and loop plugins without the empty="true" attribute. This can lead to memory consumption issues.

Inefficient code example:

<while condition="${i <= 1000}" index="i">
<!-- some calculation logic with i incrementation -->
</while>

Recommended approach. Almost in all cases, add the empty attribute with the true value.

<while condition="${i <= 1000}" index="i" empty="true">
<!-- some calculation logic with i incrementation -->
</while>

Advanced approach. Apply the approach only when you assign or use the output of the plugin execution. The example shows when the empty attribute should not be present or should have the false value:

<export include-original-data="false">
<single-column name="key" value='${value}'/>
<loop item="fieldName">
    <list>
        <var name="fieldList"/>
    </list>
    <body>
        <single-column name="${fieldName}" value='${resultMap.getWrappedObject().get(fieldName.toString())}'/>
        </body>
</loop>
</export>

In the example above, the empty attribute is not used because the output of the loop plugin is utilized in the export plugin.

call​

case​

config​

database​

db-param​

empty​

exit​

file​

ftp​

function​

html-to-xml​

json-to-xml​

loop​

mail​

mail-attach​

regexp​

return​

script​

template​

text​

tokenize​

try-catch​

var​

var-def​

while​

xml-to-json​

xpath​

xslt​

xquery​

zip​

Optimize memory usage when designing Bot Tasks​

call

case

config

database

db-param

empty

exit

file

ftp

function

html-to-xml

json-to-xml

loop

mail

mail-attach

regexp

return

script

template

text

tokenize

try-catch

var

var-def

while

xml-to-json

xpath

xslt

xquery

zip

Optimize memory usage when designing Bot Tasks