9.1

Edit

Optimize memory usage

There are some common mistakes in Bot Task design which can lead to memory consumption issues.

Use <while> and <loop> plugins without empty="true" attribute

Example of inefficient code:

<while condition="${i <= 1000}" index="i">
    <!-- some calculation logic with i incrementation -->
</while>

Recommended approach:

<while condition="${i <= 1000}" index="i" empty="true">
    <!-- some calculation logic with i incrementation -->
</while>

Almost in all cases, you should add the empty attribute with the true value.

Advanced: You don't need it only in case you assign/use output of this plugin execution. Example when this attribute should not be present or should have the false value:

<export include-original-data="false">
    <single-column name="key" value='${value}'/>
    <loop item="fieldName">
        <list>
            <var name="fieldList"/>
        </list>
        <body>
            <single-column name="${fieldName}" value='${resultMap.getWrappedObject().get(fieldName.toString())}'/>
        </body>
    </loop>
</export>

The empty attribute is not used here because the output of the <loop> plugin is used in the <export> plugin.

Select all the data from Data Store

Example of inefficient code:

<var-def name="resultList">
    <datastore name="${someDatastore}">
        select * from @this;
    </datastore>
</var-def>
<script>
    <![CDATA[
        String name = resultList.toList().get(0).get("name").toString();
        // some actions with variable name
    ]]>
</script>

This script selects all the data from a Data Store, but only the first record is needed. As a result, a big amount of unused data is loaded into the memory.

Recommended approach: select only required rows and columns.

<var-def name="resultList">
    <datastore name="${someDatastore}">
        select name from @this LIMIT 1;
    </datastore>
</var-def>
<script>
    <![CDATA[
        String name = resultList.toList().get(0).get("name").toString();
        // some actions with variable name
    ]]>
</script>

Export large documents between steps

The Control Tower engine stores all export results in its database and loads them for the next processing step(s). Therefore, it is recommended to save large documents/JSONs to Data Stores or S3 file storage (if data are not changed on the next steps), and read them afterwards only when needed.

When using a split rule, the Control Tower engine duplicates all outcoming data for each new record. For examle, you have a 10 MB document and split its content into 100 records. In this case, the Control Tower application will need 1 Gb of RAM to successfully process the step with split data. The batch processing introduced for such cases does not eliminate the issue completely.

Export useless data between steps

The Control Tower engine stores all export results in its data base and loads them for the next processing step(s). Therefore, it is recommended not to export data which will not be used on the subsequent steps and not needed in the final snapshot.

Consider usage of the include-original-data="false" attribute in the export plugin.

← PreviousNext →