Start task and Business Process from Data Source

A Data Source is a mechanism providing input data and configuration for your Business Processes (BPs) and tasks. The input format is best suited for those use cases where the data resides on an external source, such as an SFTP Server or Amazon S3 Cloud.

To start an MT or a BP from a Data Source, you need the following:

• UUID of the existing task or BP (Definition UUID)
• Input Data file (CSV) uploaded to an external server
• Configuration file (CCF) uploaded to an external server
• Data Source: a WorkFusion object containing the external server credentials and path to the configuration file
• Schedule: the number of occurrences per each Data Source update

Create configuration file

The configuration file (CCF) contains all MT or BP parameters needed to start a new run. The file contains parameter-value pairs.

To download a sample CCF file, click the link.

The table below lists the parameters and values from a CCF file.

Parameter	Value	Description
ACTIVE	TRUE	Default parameter.
DATAFILE	input_file.csv	It is the relative path to the input CSV data file in the S3 storage, the full path inside your bucket. The server folder with the CCF file is the root.
DATAFILE_SIZE	-1	Default parameter.
FILE_SEPARATOR	COMMA; PIPE; SEMICOLON; TAB	Separator used in the input file.
ENV	DEVELOPMENT; PRODUCTION	Environment
BLOCK_SIZE	1	Block size
TASK_DISPLAY_PRIORITY	MAX; NORMAL; MIN	Task priority; tasks with the max priority are displayed first if the task type is the same.
CAMPAIGN_UUID	79eea42a-127e-4613-83a2-7fbe80ba0a38	Create an MT or BP (Campaign) in WorkFusion and copy the campaign UUID, or copy the UUID of the existing Campaign.
tags	datasource, test	List of tags.
SNAPSHOT_TYPE	XLSX; CSV	Snapshot format.
OUTPUT_DIR	results/	Relative path for storing the MT or BP snapshots. The server folder with the CCF file is the root.
STREAM_TYPE	Immediately; PermanentOpenTask	Immediately: all hits are created on the endpoint. PermanentOpenTask: a specific quantity is displayed. If the value is not specified, the setting defaults to Immediately.
STREAM_VALUE	10	Works if STREAM_TYPE=PermanentOpenTask. The default value is 10.
STREAM_THRESHOLD	1 (if an absolute value); 25% (if percentage)	The setting defaults to 100% if the value is not specified.

Upload files to server

Upload the input file (CSV) and configuration file (CCF) to an external server (FTPS, S3, SFTP).

Remember to check the file's and folder's READ and WRITE permissions on the external server.

Create Data Source

To start managing Data Sources, go to the Control Tower main menu and then navigate to System Settings > Data Sources.

On this page, you can create, copy, edit , and delete Data Sources.

To create a Data Source, follow the instruction below:

1. Click the Create Data Source button.

   

2. Enter the appropriate Name and Description for your Data Source.

3. The platform automatically generates a UUID for the new Data Source. You can use this UUID in API calls or schedules.

4. Select a Type from the appropriate drop-down box:

   • S3: Amazon's Cloud Server
   • FTPS: File Transfer Protocol + SSL encryption
   • SFTP: Secure File Transfer Protocol based on SSH

5. Enter Configuration Parameters that depend on the connection type:

   • S3

     • accessKey: Access Key ID

     • secretKey: Secret Access Key

     • bucket: destination bucket in S3

     • keyPrefix: remote folder

       Note that keyPrefix should end with the slash symbol (/).

   • SFTP

     • host: hostname to establish a connection over the SFTP protocol
     • port: port number to connect to on the SFTP server
     • login: SFTP Server account user name
     • password: SFTP Server account password
     • remote_folder: directory on the SFTP server to start the session
     • private_key: key stored on your local machine to enable authentication without password
     • public_key: key stored on the SFTP server to enable authentication without password

   • FTPS

     • host: hostname to establish a connection over the FTPS protocol
     • port: port number to connect to on the FTPS server
     • login: FTPS server account user name
     • password: FTPS server account password
     • remote_folder: directory on the FTPS server to start the session

6. Click Save to save the Data Source.

Create schedule

To create a schedule for BP or MT runs, follow the steps below:

1. Go to Advanced > Schedules, click the Create button.

   

2. On the Create Schedule page, find the Task or Process Definition drop-down box and select your BP or task.

   

3. In the Input Data drop-down box, choose Data Source. In the field that appears next to the drop-down box, select the Data Source you created.

4. Set other schedule parameters as required. Click Save.

After the schedule is created, the specified task or BP starts automatically. The platform polls the Data Source for any input files at a specified interval (for instance, every 10 minutes). If such a file exists, the platform runs it.

View result

To verify that your task or BP started, log in to the external server and open the folder with the CCF file. Verify that {name}.ccf.uploaded file was created.

When a task or BP is completed, the following happens:

• The {name}.ccf.done file appears.
• A result file is created in the appropriate folder (OUTPUT_DIR parameter of the CCF file).

tip

If a Bot Task is not started, make sure that the Bot Configuration for the step doesn't contain the Bot Source value with Availability options.