Java Native Worker · WorkFusion Documentation

The Java native mechanism enables you to write and deploy custom pure Java-based Workers with any libraries without additional loading or linking.

Creating a Java Native Worker consists of the following stages:

Generate a Worker application from the archetype.
Add necessary modules.
Implement task functionality in the form of Java classes. Add needed third-party dependencies if required.
Build and test an implementation locally through unit tests.
Create a Business Process (BP) with Java Worker tasks.
Test the BP. You can do it in one of the following ways:
- Deploy the Worker to Control Tower (CT).
- Run the Worker from IDE in the debug mode.
Modify, update, re-deploy, or test Workers as needed.

Generate Worker from archetype

We have created the java-native-worker-archetype Maven archetype to build a new Worker application.

To generate a Worker application from the archetype, run the following command in the parent directory:
```
mvn archetype:generate -DarchetypeGroupId=com.workfusion.spa.core -DarchetypeArtifactId=java-native-worker-archetype -DarchetypeVersion=X.X.X.X
```
It can be the root directory of a Digital Worker (DW) project or an empty directory if you want to develop your Worker as a standalone project.

Provide the following properties when the command prompt appears:

groupId: Worker's Maven group ID.
artifactId: Worker's Maven artifact ID. Also, it is the default directory and the target artifact name.
version: Worker's Maven version.
package: root package for Worker Java classes.

Define value for property 'groupId': com.mycompany
Define value for property 'artifactId': my-test-worker
Define value for property 'version' 1.0-SNAPSHOT: :
Define value for property 'package' com.mycompany: : com.mycompany.worker
Confirm properties configuration:
groupId: com.mycompany
artifactId: my-test-worker
version: 1.0-SNAPSHOT
package: com.mycompany.worker
Y: : y

A new project appears in a sub-directory with the name taken from the artifactId value. The new project is the pre-configured spring-boot application with minimum dependencies required to implement the base Worker logic and a sample task processor.

Add modules

Only the module-event module that provides the "send the event to CT" component is default added as a dependency in the generated application.

In frequent cases, you might need the following modules:

module-secrets to access secrets from Vault.

note

By default, you can access configuration parameters in Vault using standard spring mechanisms (@Value annotation, and so on), and you don't need to access this module. It is required only to access custom secrets defined in the CT interface.
module-billing to send billing information from a step.

See details on each module below:

module-event

module-secrets

module-billing

module-distributed-services

Event Module is a project that defines all necessary classes and APIs used to log events from the Java Native Worker into BEP.

Use the Maven tool for building.

For a regular build, run the following command:
```
mvn clean install -T 1C
```
To build the project without tests, run the following command:
```
mvn clean install -Dmaven.test.skip=true -T 1C
```

To apply the module, do as follows:

Add a dependency to your project's pom.xml. Note that for projects created using the Java Native Worker archetype, this module is already included in pom.xml.

<dependencies>
    <dependency>
        <groupId>com.workfusion.spa.java.native.worker</groupId>
        <artifactId>module-event</artifactId>
    </dependency>

Inject the event logger into the component:

import org.slf4j.Logger;
import com.workfusion.spa.jnative.worker.module.event.EventLogger;

@TaskProcessor(id = "sample-task")
public class SampleTaskProcessor implements ITaskProcessor {

    private final Logger logger;

    @Autowired
    public SampleTaskProcessor(EventLogger workerLogger) {
        this.logger = workerLogger.getEventLogger();
    }

    @Override
    public TaskOutputData process(TaskInputData input) throws TaskProcessorException {
        try {
            // ...
            //send log by worker logger into Control Tower as an event.
            logger.info("Some event info for CT side.");
            // ...
        } catch (Exception e) {
            //send error event to CT and re-throw exception
            logger.error(e.getMessage(), e);
            throw e;
        }
    }
}

Secrets Module is a project that defines all needed classes and APIs used for getting secrets from HashiCorp Vault.

Use the Maven tool for building.

For a regular build, run the following command:
```
mvn clean install -T 1C
```
To build the project without tests, run the following command:
```
mvn clean install -Dmaven.test.skip=true -T 1C
```

To apply the module, do as follows:

In your Java Native Worker, add or uncomment the required dependencies. In the example below, you can uncomment the <module-secrets> dependency if a module that works with client secrets is needed.

<dependencies>
    <dependency>
        <groupId>com.workfusion.spa.java.native.worker</groupId>
        <artifactId>module-event</artifactId>
    </dependency>

    <dependency>
        <groupId>com.workfusion.spa.java.native.worker</groupId>
        <artifactId>module-secrets</artifactId>
    </dependency>
</dependencies>

Check that the secure.storage.safe.customer.default property is defined in the ZooKeeper context. This value configured during the installation procedure must be the same as in CT.

See the example of using the secret service:

import java.util.Optional;

import org.slf4j.Logger;
import org.springframework.beans.factory.annotation.Autowired;

import com.workfusion.spa.jnative.worker.core.api.TaskInputData;
import com.workfusion.spa.jnative.worker.core.api.TaskOutputData;
import com.workfusion.spa.jnative.worker.core.api.TaskOutputRow;
import com.workfusion.spa.jnative.worker.core.task.ITaskProcessor;
import com.workfusion.spa.jnative.worker.core.task.TaskProcessor;
import com.workfusion.spa.jnative.worker.core.task.TaskProcessorException;
import com.workfusion.spa.jnative.worker.module.event.EventLogger;
import com.workfusion.spa.jnative.worker.module.secret.SecretService;
import com.workfusion.spa.jnative.worker.module.secret.entity.SecretDto;

@TaskProcessor(id = "secret-task")
public class SecretTaskProcessor implements ITaskProcessor {

    //Java Native Worker logger, all messages will be sent into CT as an event.
    private final Logger logger;

    //Java Native Worker client secret service.
    private final SecretService secretService;

    @Autowired
    public SecretTaskProcessor(EventLogger workerLogger, SecretService secretService) {
        this.logger = workerLogger.getEventLogger();
        this.secretService = secretService;
    }

    @Override
    public TaskOutputData process(TaskInputData input) throws TaskProcessorException {
        try {
            // create client secret with provided data.
            secretService.createSecret("my-custom-identifier", "secret-name", "secret-value");
            // update client secret, which will be created above.
            secretService.updateSecret("my-custom-identifier", "updated-name", "updated-value");
            // reset secret value for provided alias.
            secretService.resetSecret("my-custom-identifier");
            // get client secret with provided alias.
            Optional<SecretDto> optionalSecret = secretService.getSecret("my-custom-identifier");
            // delete client secret for provided alias
            secretService.deleteSecret("my-custom-identifier");

            //task processing logic

            //build output
            return new TaskOutputData()
                    .addRow(new TaskOutputRow()
                            //.put("some_field", "some_value")
                            //.put("other_field", "other_value")
                    )
                    //add more result row if required
            ;
        } catch (Exception e) {
            logger.error(e.getMessage(), e);
            throw e;
        }
    }

}

Billing Module is a project that defines all necessary classes and APIs used to log billing information from the Java Native Worker into BEP.

Use the Maven tool for building.

For a regular build, run the following command:
```
mvn clean install -T 1C
```
To build the project without tests, run the following command:
```
mvn clean install -Dmaven.test.skip=true -T 1C
```

To apply the module, do as follows:

Add a dependency to your project's pom.xml. Note that for projects created using the Java Native Worker archetype, this module is already included in pom.xml:
```
<dependency>
    <groupId>com.workfusion.spa.java.native.worker</groupId>
    <artifactId>module-billing</artifactId>
</dependency>
```

Inject the billing service into the component:

import org.slf4j.Logger;
import com.workfusion.spa.jnative.worker.module.event.EventLogger;

@TaskProcessor(id = "sample-task")
public class SampleTaskProcessor implements ITaskProcessor {

    private final BillingInfoService billingInfoService;

    @Autowired
    public SampleTaskProcessor(...., BillingInfoService billingInfoService) {
        // ...
        this.billingInfoService = billingInfoService;
    }

    @Override
    public TaskOutputData process(TaskInputData input) throws TaskProcessorException {
        String billingInfoJson = // ... build billing info and convert to JSON string
        billingInfoSerivce.sendInfo(billingInfoJson);
    }
}

Distributed Service Module contains a service for working with the distributed object pool (analog to the WebHarvest pool plugin).

Use the Maven tool for building.

For a regular build, run the following command:
```
mvn clean install -T 1C
```
To build the project without tests, run the following command:
```
mvn clean install -Dmaven.test.skip=true -T 1C
```

To build and deploy the module into Nexus, use the https://jenkins.workfusion.com/job/Java-Native-Worker/job/java-native-worker-module-distributed-services/ job.

To apply the module, do as follows:

Add a dependency to your project's pom.xml. Note that for projects created using the Java Native Worker archetype, the dependency is already present in pom.xml. In this case, uncomment it.
```
<dependency>
    <groupId>com.workfusion.spa.java.native.worker</groupId>
    <artifactId>module-distributed-services</artifactId>
</dependency>
```

Inject the object pool service into your processor.

import com.workfusion.spa.jnative.worker.module.distributed.services.impl.ObjectPoolRunnerFactory;

@TaskProcessor(id = "sample-task")
public class SampleTaskProcessor implements ITaskProcessor {

    private final Logger logger;
    private final ObjectPoolRunnerFactory objectPoolRunnerFactory;

    @Autowired
    public SampleTaskProcessor(EventLogger workerLogger, ObjectPoolRunnerFactory objectPoolRunnerFactory) {
        this.logger = workerLogger.getEventLogger();
        this.objectPoolRunnerFactory = objectPoolRunnerFactory;
    }

    @Override
    public TaskOutputData process(TaskInputData input) throws TaskProcessorException {
        try {

            //Define a supplier that provides a list of shared objects
            List sharedObjects = Arrays.asList("a", "b");
            Supplier<List<Object>> sharedObjectSupplier = () -> sharedObjects;

            //Define a function to process a shared object
            Function<Object, Object> sharedObjectProcessor = o -> {
                try {
                    Thread.sleep(2000L);
                } catch (InterruptedException e) {
                    e.printStackTrace();
                }
                logger.warn("OBJECT FROM POOL " + o);
                doSomethingWithSharedObject(o);
                return null;
            };

            //Create and call objectPoolRunner and pass the shared object supplier and the processor
            ObjectPoolRunner objectPoolRunner = objectPoolRunnerFactory.createObjectPoolRunner("test_pool_id");
            objectPoolRunner.executeUsingObjectPool(sharedObjectSupplier, sharedObjectProcessor);

            //Build output
            return new TaskOutputData()
                    .addRow(new TaskOutputRow()
                            .put("some_field", "some_value")
                            .put("other_field", "other_value")
                    );
        } catch (Exception e) {
            logger.error(e.getMessage(), e);
            throw e;
        }
    }

You can also mock ObjectPoolProcessor in your test context Worker.

class SampleTaskProcessorTest extends BaseProcessorTest {

    @Autowired
    TaskProcessingRouter router;

    //It's mock
    @Autowired
    ObjectPoolProcessor objectPoolProcessor;

    @DisplayName("Simple Task Processor generated test case")
    @Test
    void simpleTaskProcessorTest() {
        MySerializableObject object = new MySerializableObject();
        
        when(objectPoolProcessor.borrowObject("any-string", Arrays.asList(object))).thenReturn(object);
        ....
    }
}

The modules are added to pom.xml but commented. To use them, uncomment the corresponding lines:

<dependencies>
    <dependency>
        <groupId>com.workfusion.spa.java.native.worker</groupId>
        <artifactId>module-event</artifactId>
    </dependency>

    <!--uncomment next dependency, if module for sending billing info is needed-->
    <!--<dependency>
        <groupId>com.workfusion.spa.java.native.worker</groupId>
        <artifactId>module-billing</artifactId>
    </dependency>-->

    <!--uncomment next dependency, if a module, which works with client secrets, is needed-->
    <!--<dependency>
        <groupId>com.workfusion.spa.java.native.worker</groupId>
        <artifactId>module-secrets</artifactId>
    </dependency>-->
</dependencies>

You can add other modules manually in the same way as you add any other Maven dependencies:

<dependencies>
    <!-- ... -->

    <dependency>
        <groupId>com.workfusion.spa.java.native.worker</groupId>
        <artifactId>module-with-some-functionality</artifactId>
        <version>1.2.3</version>
    </dependency>
</dependencies>

Implement task functionality

To implement the Worker functionality, add the ITaskProcessor classes: one class per each task type.

In the generated application, the archetype creates SampleTaskProcessor in the task sub-package. You can use SampleTaskProcessor as an example or a starter for task processors. This sample task already contains an injected logger for sending events to CT.

@TaskProcessor(id = "sample-task")
public class SampleTaskProcessor implements ITaskProcessor {

    //Java Native Worker logger, all messages will be sent into CT as an event.
    private final Logger logger;
    
    @Autowired
    public SampleTaskProcessor(EventLogger workerLogger) {
        this.logger = workerLogger.getEventLogger();
    }

    @Override
    public TaskOutputData process(TaskInputData input) throws TaskProcessorException {
        try {

            //task processing logic

            //send log by worker logger into Control Tower as an event.
            logger.info("Some event info for CT side.");


            //build output
            return new TaskOutputData()
                    .addRow(new TaskOutputRow()
                            //.put("some_field", "some_value")
                            //.put("other_field", "other_value")
                    )
                    //add more result rows if required
            ;
        } catch (Exception e) {
            logger.error(e.getMessage(), e);
            throw e;
        }
    }

}

A Task processor is a spring component, so you can use any spring mechanism in it, for example:

Auto-wiring
Post-construct and per-destroy
Access environment and configuration, and so on

Build and test locally

Use standard practices for writing and running unit or integration tests for your Workers.

You can build the Worker with the standard maven mvn clean install command. After building it, you get a spring-boot application artifact in the /target directory. You can deploy the artifact to a target environment as any other Worker's artifacts.

Create BP with Java Worker tasks

To add tasks to the Java Native Worker, perform the following steps:

Create a BP.
Add a Bot Task and edit its source:
1. Add type="java" to the root <config> tag.
2. Remove everything inside <config> and populate the tag with the Java native task metadata.
Save the step.
Save the BP.

The Java Native Worker tasks have the minimum structure:

<?xml version="1.0" encoding="UTF-8"/>
<config type="java">
    <worker>${GAV}</worker>
    <processor>${processorId}</processor>
</config>

Here, replace:

${GAV} with your Worker's current groupId:artifactId:version. You can find this parameter in pom.xml.
${processordId} with the corresponding task processor ID. See the id attribute in the @TaskProcessor annotation in a corresponding class.

For example, groupId:artifactId:version ($GAV) in your Worker's pom.xml looks as follows:

<project>
    <groupId>my.comp</groupId>
    <artifactId>my-worker</artifactId>
    <version>1.2.3</version>
    ...

You create two task processors:

@TaskProcessor(id = "first-task-type")
public class FirstTaskProcessor implements ITaskProcessor { ... }

@TaskProcessor(id = "second-task-type")
public class SecondTaskProcessor implements ITaskProcessor { ... }

In this case, the tasks are as follows:

For FirstTaskProcessor:

<?xml version="1.0" encoding="UTF-8"/>
<config type="java">
    <worker>my.comp:my-worker:1.2.3</worker>
    <processor>first-task-type</processor>
</config>

For SecondTaskProcessor:

<?xml version="1.0" encoding="UTF-8"/>
<config type="java">
    <worker>my.comp:my-worker:1.2.3</worker>
    <processor>second-task-type</processor>
</config>

You can use the following Task parameters:

Parameter	Required	Description
`worker`	Yes	`groupId:artifactId:version` of the Java Native Worker artifact.
`processor`	Yes	Task processor ID. The code is used to find a processor class and route a task to it.
`splitData`	No	Parameter that defines the split-data behavior. Values: `auto` (default): automatically detects split behavior: If the result contains a single record, do not split. If the result contains two or more records, split data `force`: always split data. Applicable for the monitoring-loop flow, and so on.
`sendResultToCaller`	No	Parameter that defines if CT must send results to an external connector (for example, REST connector), a caller BP, and so on. Implements the same logic as the `send-to-external-connector` attribute in `export-plugin`. Values: `false` (default): do not try to send results from this step. `true`: try to send results from this step to an external connector, a caller BP, and so on.

Run BP to test logic

Deploy Worker to CT

To test the Java Native Worker, deploy it to a target environment and run the corresponding BP. You can deploy the Worker by importing the artifact as a part of the DW package.

Run Worker from IDE in debug mode

You can run the Worker manually to debug it in IDE. In this case, provide the minimum set of configuration parameters.

Full environment

To run the Worker against a full environment, do the following preparations and preconditions checks:

Make sure the Worker is not deployed to Nexus on the environment to prevent WMS from starting on BEP cluster Worker instances that "steal" tasks from your locally running Worker. If the Worker is already deployed on Nexus, you must perform one of the following actions:
- Delete the Worker artifact from Nexus and the shared directory.
- Stop WMS and kill custom Workers (if any) in Marathon. Note that this prevents starting any Worker on a BEP cluster. Thus, if your BP has mixed step types, it will not work.
Download the client certificate /opt/workfusion/ssl/vault_workfusion.p12 for Vault to a local machine.
Copy the secure.storage.* properties from the /opt/workfusion/conf/workfusion.properties file. Add the properties to the Worker configuration parameters. Replace the value for secure.storage.client.certificate with a local path to the downloaded vault_workfusion.p12.

Run the Worker with the following parameters:

spring.cloud.zookeeper.connect-string=${master-host}:2181
execution.task.queue.input=exec.${ct-client-id}.worker.${gav}.DEFAULT

secure.storage.type=VAULT
# ... and other 'secure.storage.*' parameters

Run the BP.

Deploy Worker to IA Cloud platform

The native Java Worker, like any other Worker types (CT, AutoML, OCR), must be deployed to Nexus into the workers repository, so WMS can download and start it.

To deploy Java Native Workers to the IA Cloud platform, import the Worker artifact as a part of the DW Asset Bundle.

You can import the Java Native Worker artifact and a Bot Config, where it is used, with the help of the Asset Bundle import mechanism.

Two components of the bundle import are used: artifactory dependency import and Bot Config import.

Import artifact

Before importing the artifact, ensure that you have a JAR file with some groupId:artifactId:version names.

The following instruction is applicable for building a minimum standalone bundle from scratch. The bundle contains only a Java Native Worker artifact or for adding Worker artifact to some existing bundle with repackage (steps 3, 4, 5, 7).

To import the Worker artifact, perform the following steps:

Create the artifactory-dependency directory in the bundle if it does not exist.
In the artifactory-dependency directory, create the workers directory as the first underlying directory name. It represents the name of the Nexus repository where the artifact must be imported.
In the workers directory, create the tree of $GAV directories. The group name must be split into the hierarchy of sub-directories by a dot in the group name, for example, com.supertest.
In the version directory, place the JAR artifact named according to the convention: artifact_name-version.jar.

In the bundle's root directory (the same layer as the artifactory-dependency directory), create the meta-info.json file. See the structure and description in Package assets into DW Asset Bundle.

artifactory-dependency
│   └── workers
│   │       └── com
│   │           ├── supertest
│   │               └── test-native-worker
│   │                   └── 1.0
│   │                       └── test-native-worker-1.0.jar

In this case, Worker's GAV is com.supertest:test-native-worker:1.0.

Archive the above structure into a ZIP archive. Your bundle is now ready.

note

The workers Nexus repository is a release repository. You can't import an artifact containing the SNAPSHOT postfix in its version. The release artifact versions are required.

Import Bot Config

When importing a Bot Config, you have the following options:

Import from the Java Native Worker artifact.
Import from an Asset Bundle.

Import from Java Native Worker artifact

A Bot Config that contains the Java Native Worker processor call can be imported directly from the Java Native Worker artifact.

Place the Bot config XMLs in the src/main/resources/configs/main directory as shown below:

my-custom-worker
     └──src
          └──main
               └──resources
                      └──configs
                           └──main
                                ├──email_intake.xml
                                └──validation.xml

When you build the Java Native Worker and import it as a part of the Asset Bundle, machine configs are also imported to CT.

You can only import machine configs that call to the Java Native Worker processors in such a way. This approach makes importing WebHarvest or some arbitrary Bot Config Bundle impossible.

Import from Asset Bundle

A Bot Config (step) configured to use Java Native Worker processing can be imported similarly to any other.

Before importing a Bot Config from the Asset Bundle, ensure you have the bot-config.xml file exported previously from another instance.

To import a Bot Config, do as follows:

Create the bot-config directory if it doesn't exist in a bundle.
Place the XML bot config file in the bot-config directory.
Create the meta-info.json file as described in the previous section.
Archive the above files and directories into the ZIP archive. Your bundle is now ready.
```
bot-config
│   └── some-config.xml
```

See the example in some-config.xml.

important

The config contains UUIDs, names, and other components taken from the original instance. If you manually modify its machine configuration in a text editor or export the same step with different machine code configurations without changing UUIDs, the bot config machine code will be overwritten during the import if an existing configuration is present with identical IDs.

The bundle structure with the Java Native Worker and a bot configuration looks as follows:

bot-config
│   └── some-config.xml
artifactory-dependency
│   └── workers
│   │       └── com
│   │           ├── supertest
│   │               └── test-native-worker
│   │                   └── 1.0
│   │                       └── test-native-worker-1.0.jar