Version: 10.3

S3 plugins

s3

The plugin lets you access and manage data on the Amazon S3 server.

View details

The s3 plugin wraps all S3 calls and keeps the connection. For the purpose, workfusion.properties uses the following properties respectively:

s3.endpoint.url
s3.bucketName

When running examples, create an S3 bucket and use its name in scripts, for example, s3 bucket="my_bucket_name".

As S3 credentials contain sensitive information, they are stored in Secrets Vault and defined in the following secure properties:

s3.access-key
s3.secret-key

For details on how to configure access to S3, refer to Upload files to Amazon AWS S3 | Configure public access to file or directory.

The plugin contains the following attributes:

Name	Required	Description
`bucket`	Yes	S3 bucket name.
`access-key`	No	S3 access-key value.
`secret-key`	No	S3 secret-key value.
`s3-keys-type`	No	S3 access and secret key type.
`endpoint-url`	No	Used to override the defined endpoint by the value from the parent S3 plugin or `s3.endpoint.url` in `workfusion.properties`.

The plugin returns a list containing instance(s) of the com.freedomoss.crowdcontrol.webharvest.plugin.s3.S3ResultItem class (one per uploaded file).

caution

You can override S3 credentials in the plugin, but avoid doing so on production instances.

For s3-keys-type, you can map an access key, a secret key, and an S3 URL to a certain context where the key is used. The mapping is stored in Secrets Vault, as shown below for the sfs and vds type:

s3.context.key.map : {"sfs":{"endpoint-url":"https://s3.amazonaws.com", "secret-key":"123Abc","access-key":"123Abc"},"vds":{"secret-key":"123Abc","access-key":"123Abc"}}

To apply these properties in the plugin, use the s3-keys-type attribute, for example:

<s3 bucket="my_bucket_name" s3-keys-type="vds">
    ...
</s3>

This allows you to do the following:

To store specific data on a specifically designed S3 instance
To switch across multiple VDS instances in a Business Process

s3-get

The plugin reads a single object (a file) from S3.

View details

The s3-get plugin contains the following attributes:

Name	Required	Default	Description
`name`	Yes		Object name in S3.
`type`	No	`auto`	Returns an S3 object type: `binary`, `string`, `auto`.
`decryption-key`	No	-	Key for decrypting the received content. Must be a valid 256-bit Base64-encoded string.

Example without a decryption key:

<s3 bucket="my_bucket_name">
    <var-def name="content">
        <s3-get name="path_to-file/file.txt"/>
    </var-def>
</s3> 

Example with a decryption key:

<?xml version="1.0" encoding="UTF-8"?>
<config charset="UTF-8">
        <secrets-vault-get alias="my-encryption-key" />

        <script><![CDATA[
                Map entryMap = secureEntryMap.getWrappedObject();
                com.workfusion.bot.service.SecureEntryDTO obj = entryMap.get("my-encryption-key"");
                encryption_key = obj.getValue().toString();
        ]]></script>

    <script><![CDATA[
       uuid = java.util.UUID.randomUUID();
       uuidCopy = java.util.UUID.randomUUID();
       bucket = "automation-tests";
    ]]></script>

    <s3 bucket="${bucket}">
        <var-def name="read_content">
            <s3-get name="copy/decrypt_encrypted/${uuidCopy}.txt"
                    decryption-key="${encryption_key}" />
        </var-def>
    </s3>

    <export include-original-data="true">
         <single-column name="read_content" value="${read_content}" />
    </export>
</config>

s3-list

The plugin gets a list of objects, such as files or folders, from Amazon S3.

View details

The s3-list plugin supports filtering by the RegExp filename pattern and dates in the dd/MM/yyyy hh:mm:ss aaa z format. It contains the following attributes:

Name	Required	Default	Description
`pattern`	No	`.*`	RegExp pattern to match against a remote S3 location.
`min-date`	No		Minimum date for the last modified date in the dd/mm/yyyy hh:mm:ss aaa z format.
`max-date`	No		Maximum date for the last modified date in the dd/mm/yyyy hh:mm:ss aaa z format.
`prefix`	No		Optional parameter that restricts the response to keys beginning with a specified prefix.
`recursive`	No	`true`	Two values are possible: `true` or `false`. If the value is `true`, the plugin lists all files under all folders for a specified bucket and prefix (if any).

note

Setting recursive to true can affect performance. If you use recursive=true, the recommended practice is to combine it with the prefix parameter that contains the name of the folder to be analyzed by the plugin.

Example 1

<s3 bucket="temp-bucket">
        <var-def name='result'>
            <s3-list recursive="true" prefix="under_folder/"/>
        </var-def>
    </s3>

Example 2

<s3 bucket="temp-bucket">
        <var-def name='result'>
            <s3-list recursive="false" prefix="under_folder/"/>
        </var-def>
    </s3>

<s3 bucket="temp-bucket">
        <var-def name='result'>
            <s3-list recursive="false"/>
        </var-def>
    </s3>

Example 3

<s3 bucket="temp-bucket">
        <var-def name='result'>
            <s3-list prefix="under_folder/"/>
        </var-def>
    </s3>

s3-delete

The plugin deletes a single object, such as a file or a folder, from S3.

View details

The s3-delete plugin contains the following attributes:

Name	Required	Default	Description
`name`	Yes		Object name in S3

Example:

<s3 bucket="my_bucket_name">
    <var-def name="content">
        <s3-delete name="path_to-file/file.txt"/>
    </var-def>
</s3> 

s3-put

The plugin uploads files to Amazon S3.

View details

It is strongly recommended to provide a charset within content-type, for example, content-type="text/html; charset=utf-8". Otherwise, the default HTML 4.0 ISO-8859-1 is used.

The s3-put plugin is asynchronous: the underlying put operation takes place only after the s3 plugin ended. Do not mistake the logic for the one relying on the synchronous put inside a single s3 block.

After uploading, you can access files via an Amazon S3 link. The retry policy is up to three retries of IOException, 5xx error code, and a throttling error response.

The plugin contains the following attributes:

Name	Required	Default	Description
`path`	Yes		Full filename, including the path to the Amazon S3 bucket.
`acl`	No	`AwsExecRead`	Access control list; possible values are `AwsExecRead`, `Private`, `PublicRead`, `PublicReadWrite`, `AuthenticatedRead`, `LogDeliveryWrite`, `BucketOwnerRead`, `BucketOwnerFullControl`. For more information, refer to the AWS API documentation.
`content-type`	No		Content type of the uploaded file (pass to Amazon S3).
`content-disposition`	No		Content disposition of the uploaded file (pass to Amazon S3).
`expires-in-seconds`	No	No signed URL generated.	Period of time in seconds for which the file's signed URL is live. The signed URL is stored in the `signedUrl` property of the returned object. You can access it via the `getSignedUrl()` method or the property getter (see Example 1 below).
`encryption-key`	No	-	Key for encrypting the received content. Must be a valid 256-bit Base64-encoded string.

Example 1:

<config>
  <var-def name="content">
      <http url="${url}"/>
  </var-def>
  <var-def name="s3Link">
    <s3 bucket="${bucket}">
      <s3-put path="training-set/${java.util.UUID.randomUUID()}.txt" acl="Private" expires-in-seconds="3600" content-type="text/html" content-disposition="inline">
        <var name='content'/>
      </s3-put>
    </s3>
  </var-def>
  <export include-original-data="true">
    <multi-column list="${s3Link}">
      <put-to-column-getter name="direct-url" property="directUrl"/>
      <put-to-column-getter name="signed-url" property="signedUrl"/>
      <put-to-column-getter name="filename" property="filename"/>
    </multi-column>
  </export>
</config>

Example 2:

<config>
    
    <var-def name="content">
        <http url="http://example.com"/>
    </var-def>
      
    <s3 bucket='temp_bucket'>
        <s3-put path="example.com.file" content-type="text/html; charset=utf-8" content-disposition="inline" acl="PublicRead">
            <var name="content"/>
        </s3-put>
    </s3>
    <s3 bucket='temp_bucket'>
        <var-def name="stored">
            <s3-get name="example.com.file"/>
        </var-def>
        
        <s3-delete name="example.com.file"/>
        
        <!-- should be empty -->
        <var-def name="deleted">
            <s3-get name="example.com.file"/>
        </var-def>"
    </s3>
    
</config>

Example 3 with an encryption key:

<?xml version="1.0" encoding="UTF-8"?>
<config charset="UTF-8">
        <secrets-vault-get alias="my-encryption-key" />

        <script><![CDATA[
                Map entryMap = secureEntryMap.getWrappedObject();
                com.workfusion.bot.service.SecureEntryDTO obj = entryMap.get("my-encryption-key"");
                encryption_key = obj.getValue().toString();
        ]]></script>

    <script><![CDATA[
       uuid = java.util.UUID.randomUUID();
       uuidCopy = java.util.UUID.randomUUID();
       bucket = "automation-tests";
    ]]></script>
   <s3 bucket="${bucket}">
        <s3-put path="copy/decrypt_encrypted/${uuid}.txt" acl="PublicReadWrite" content-type="text/plain"
                encryption-key="${encryption_key}">
            <template>${content}</template>
        </s3-put>
    </s3>

    <export include-original-data="true">
         <single-column name="read_content" value="${read_content}" />
    </export>
</config>

s3-copy

The plugin copies a single object or folder to another folder or file from S3.

View details

The s3-copy plugin contains the following attributes:

Name	Required	Default	Description
`from`	Yes		File path or folder; must end with `/`.
`to`	Yes		File path or folder; must end with `/`.
`acl`	No		Access control policy for new buckets or objects. Possible values are `Private`, `PublicRead`, `PublicReadWrite`, `AuthenticatedRead`, `LogDeliveryWrite`, `BucketOwnerRead`, `BucketOwnerFullControl`. For more information, refer to the Amazon S3 documentation.
`encryption-key`	No	-	Key for encrypting the received content. Must be a valid 256-bit Base64-encoded string.
`decryption-key`	No	-	Key for decrypting the received content. Must be a valid 256-bit Base64-encoded string.

caution

Do not copy a folder to a file as the plugin throws an exception.

Example 1:

<!-- Copy file to folder-->
<s3 bucket="public.crowdcontrol">
    <s3-copy from="t/test-s3-copy-from/five_texts.csv" to="t/test-s3-copy-to/" acl="Private"/>    
</s3>

<!-- Copy file to another file --> 
<s3 bucket="public.crowdcontrol">
    <s3-copy from="t/test-s3-copy-from/five_texts.csv" to="t/test-s3-copy-to/1_five_texts.csv" acl="Private"/>    
</s3>

<!-- Copy folder (all files from folder) to folder -->
<s3 bucket="public.crowdcontrol">
    <s3-copy from="t/test-s3-copy-from/" to="t/test-s3-copy-to/" acl="Private"/>    
</s3>

Example 2 with encryption and decryption keys:

<?xml version="1.0" encoding="UTF-8"?>
<config charset="UTF-8">
        <secrets-vault-get alias="my-encryption-key" />

        <script><![CDATA[
                Map entryMap = secureEntryMap.getWrappedObject();
                com.workfusion.bot.service.SecureEntryDTO obj = entryMap.get("my-encryption-key"");
                encryption_key = obj.getValue().toString();
        ]]></script>

    <script><![CDATA[
       uuid = java.util.UUID.randomUUID();
       uuidCopy = java.util.UUID.randomUUID();
       bucket = "automation-tests";
    ]]></script>
    <s3 bucket="${bucket}">
        <s3-copy from="copy/decrypt_encrypted/${uuid}.txt" acl="PublicReadWrite" to="copy/decrypt_encrypted/${uuidCopy}.txt"
                 decryption-key="${encryption_key}" encryption-key="${encryption_key}">
            <template>${content}</template>
        </s3-copy>
    </s3>

    <export include-original-data="true">
         <single-column name="read_content" value="${read_content}" />
    </export>
</config>

Generate keys

The following S3 plugins can use optional keys to encrypt or decrypt content:

s3-get
s3-put
s3-copy

To generate the keys, run the following commands:

Bash
Groovy

cat /dev/urandom | head -c 32 | base64

<script><![CDATA[
    import java.util.Random;
    import org.apache.commons.codec.binary.Base64;

    def String generateKey(int lengthInBytes) {
        byte[] r = new byte[lengthInBytes];
        Random rnd = new Random();
        rnd.nextBytes(r);
        return Base64.encodeBase64String(r);
    }
    encryptionKey = generateKey(32);
]]></script>

s3​

s3-get​

s3-list​

s3-delete​

s3-put​

s3-copy​

Generate keys​

s3

s3-get

s3-list

s3-delete

s3-put

s3-copy

Generate keys