10.2.1

Edit

Multi-process Use Case design

Data flow

Similar to ODF 1, ODF 2 is also based on the concept of Transactions—a set of related data assemblies. No matter of data variations in any specific Use Case, a Transaction represents some entity that enters the automated workflow and needs to be processed. Processing is done by Business Processes—essentially conveyor belts that sequentially move Transactions through workstations that are Bot Tasks. Once put onto the conveyor, the Transaction travels through it until the end.

To continue the metaphor of an assembly line, you can have multiple conveyors, each dedicated to a different group of tasks, and a robot that constantly looks for free Transactions to put on the belt for each conveyor. This robot is a special kind of a Bot Task called a Monitor Task. It makes decisions based on a label that you put onto each Transaction—a Transaction Status. Any Bot Task that works with the Transaction can change the Transaction Status anytime.

If the Transaction reaches the end of the Business Process, and no Monitor Task claims it to put into another belt, it stays in the Data Store.

Monitor Tasks

A Monitor Task is a special kind of a Bot Task to put at the beginning of a Business Process. Once the Business Process starts, the Monitor Task regularly wakes up. Each time it wakes up, it can pass some Transactions to the next steps of its Business Process. These Transactions can be found in the Data Store by their Statuses or freshly created from some external data—that's how all the data appears in the system in the first place.

To achieve this, you can use a couple of tricks. One is that all users should be aware that on each run, a Monitor Task emits not only Transactions (if any) but also a special loop record. After each Monitor Task, there must be a Rule that redirects this record to the Monitor Task itself so that the monitor can run again. A loop record can be easily recognized by the _sys_repeat column with the true value.

tip

See further implementation details in the ODF 2 sources, starting with the AbstractMonitorTaskRunner class.

Configure monitor

You can configure monitors by records in the monitor_configuration Data Store. For each class of a Monitor Task, there is a configuration record. If there is none, the default configuration is assumed.

monitor_configuration Data Store

<createTable tableName="uc_some_usecase_monitor_configuration_v0">
    <column defaultValueComputed="NEWID()" name="uuid" type="NVARCHAR(36)">
        <constraints nullable="false" unique="true"/>
    </column>
    <column name="monitor_class" type="NVARCHAR(256)">
        <constraints nullable="false" unique="true"/>
    </column>
    <column name="polling_interval" type="NVARCHAR(256)"/>
    <column name="sleep_time" type="NVARCHAR(256)"/>
    <column name="max_loops" type="int"/>
    <column name="max_running_time" type="NVARCHAR(256)"/>
</createTable>

The monitor behavior is defined by four parameters:

  • Polling Interval defines the amount of time that a Monitor Task leaves between consequent polls for data. By default, the interval is equal to 60 seconds.

  • Sleep Time defines an interval after which the monitor wakes and checks if Polling Interval is elapsed.

    note

    Currently, this interval cannot be less than 60 seconds because of a Control Tower limitation.

  • Max Loops is a number of polls after which the monitor stops working. By default, the number of polls is unlimited.

  • Max Running Time defines a time period after which the monitor finishes execution. By default, the monitor execution time is unlimited.

important

Specify the polling_interval, sleep_time, and max_running_time parameters in the ISO-8601 duration format. For example, "P1D" means one day, and "PT1H30M10S" means 1 hour, 30 minutes, and 10 seconds.

Find the state of each running Monitor Task instance in the monitor Data Store. You can stop any monitor manually by setting a stopped field of the corresponding record to "1".

monitor Data Store

<createTable tableName="uc_some_usecase_monitor_v0">
    <column defaultValueComputed="NEWID()" name="uuid" type="NVARCHAR(36)">
        <constraints nullable="false" unique="true"/>
    </column>
    <column name="monitor_id" type="NVARCHAR(256)">
        <constraints nullable="false" unique="true"/>
    </column>
    <column name="creation_time" type="NVARCHAR(256)"/>
    <column name="completed_loops" type="INT"/>
    <column name="last_polling_time" type="NVARCHAR(256)"/>
    <column name="stopped" type="int"/>
</createTable>

The best practice for configuring monitors is to provide records for the monitor_configuration Data Store in database migrations. However, you can also do it manually.

In the image, you can see a record in the Data Store that configures a monitor with the EmailMonitorTask Java class. Fill the uuid field with any generated UUID as a database key. You can use this online tool.

Other fields refer to the parameters mentioned above. A monitor with this configuration tries to wake up every minute (sleep_time) and checks if 3 minutes have passed since the latest poll for data (polling_interval). It continues to run for 8 hours from its first wake-up (max_running_time) or until it has polled for data 500 times (max_loops), whatever happens first. The time interval format used is ISO-8601. Check for more samples here.

tip

See further implementation details in the ODF 2 sources, starting with the Monitor, MonitorConfigurationEntity and MonitorEntity classes.

Finalizers

A Finalizer Task is a special kind of task to stand at the end of a Business Process and change the status of each Transaction that reaches it.

Out-of-the-box cross-process lifecycle

ODF 2 offers many pre-packaged tasks designed with a specific lifecycle in mind. It is not mandatory to conform, but it is a good practice to start implementation in some default way.

Business Processes

The recommended ODF 2 implementation design provided by Full Archetype (Maven project blueprint) consists of four Business Processes:

  • Intake BP takes data from external sources, validates and sanitizes it, and otherwise prepares it to be processed.
  • Processing BP processes prepared data.
  • Submission BP is responsible for aggregating results, creating reports, and submitting to some other systems.
  • Error Processing BP handles exceptional cases raised by other BPs.

ODF 2 doesn't assume anything about actual business logic in these BPs but provides the infrastructure for them.

Transaction statuses

Default implementation expects a transaction to have one of the following statuses:

  • INTAKE_IN_PROGRESS
  • INTAKE_COMPLETED
  • PROCESSING_IN_PROGRESS
  • PROCESSING_COMPLETED
  • SUBMISSION_IN_PROGRESS
  • COMPLETED
  • HAS_ERROR
  • ERROR_IN_PROGRESS
  • ABORTED

Each ?_IN_PROGRESS status means that the Transaction is currently being processed by the corresponding BP. Each ?_COMPLETED status means the Transaction reached the end of the corresponding BP. Only Transactions in the COMPLETED status and related objects are captured in Analytics. COMPLETED and ABORTED are Transaction's terminal statuses, Transactions with these statuses cannot be modified.

Interaction of Business Processes

According to the default flow, Business Processes are arranged in a certain way:

  1. Intake Monitor polls some external data source, for example, an email account, a search engine, or a database. When it finds some data, it creates new Transactions with the INTAKE_IN_PROGRESS state and passes them further into Intake BP.

  1. When the Transaction reaches the Intake finalizer, its status is set to INTAKE_COMPLETED.

  2. Processing Monitor polls the transaction Data Store for Transactions with the INTAKE_COMPLETED status. Found Transactions are changed to PROCESSING_IN_PROGRESS and passed further into Processing BP.

  1. When the Transaction reaches the Processing finalizer, its status is set to PROCESSING_COMPLETED.

  2. Submission Monitor polls the transaction Data Store for Transactions with the PROCESSING_COMPLETED status. Found Transactions are changed to SUBMISSION_IN_PROGRESS and passed further into Submission BP.

  1. When the Transaction reaches the Submission finalizer, its status is set to COMPLETED.

With the help of Monitors and Finalizers, the Transaction can flow linearly between multiple BPs. Now, the error handling is a bit more complex.

All the out-of-the-box Task Runners are written with error handling in mind. When any exception is raised in a Transaction-related operation, its details are stored in ErrorEntity, and the Transaction status is changed to HAS_ERROR. Now, all Task Runners, including Finalizers, process only Transactions with expected status (for example, Transactions designed for Intake BP process only INTAKE_IN_PROGRESS transactions) and specifically ignore Transactions with the HAS_ERROR or ERROR_IN_PROGRESS statuses. That means that when an exception is raised, an erroneous Transaction is passed to the end of its current BP without any changes to it or its related entities.

Error Monitor specifically looks for HAS_ERROR Transactions. It finds them and passes them down to Error Handling BP. It is the only case when the Transaction can be processed by two BPs at the same time because it can take some time to reach the end of the BP it was originally in, and Error Monitor can find it faster. And that is why Transactions with error-related statuses must not be processed in any way because they can already be processed by Error Handling Tasks.

Error Handling BP is expected to analyze an error and try some form of error recovery, for example, asking human operators what to do with the help of a Manual Task. In any case, an erroneous Transaction never returns to the common data flow. At the end of the BP, its status is set to ABORTED. Error Handling BP is free to create a new Transaction with any applicable status and assign any entities from the aborted one to be processed again.

Writing Monitor Task

The simplest Monitor Task based on abstractions provided by ODF 2 looks like this:

@BotTask
public class ProcessingMonitorTask implements TransactionMonitorTask {

    @Override
    public TransactionStatus getStatusToMonitor() {
        return TransactionStatus.INTAKE_COMPLETED;
    }

    @Override
    public TransactionStatus getStatusToSet() {
        return TransactionStatus.PROCESSING_IN_PROGRESS;
    }

}

This is a real monitor for Processing BP that searches for INTAKE_COMPLETED Transactions and changes their status to PROCESSING_IN_PROGRESS. All technical stuff is hidden inside the TransactionMonitorTaskRunner and AbstractMonitorTaskRunner classes. You don't need to write this class by yourself as it is already provided by the framework. However, if you want to design your own flow, create your own analog of the TransactionMonitorTask interface and the corresponding runner.

Writing input Monitor Task

Monitors for Intake BP are much more interesting. ODF 2 has zero knowledge of how you are going to gather input data. Instead, it provides an interface to be implemented in the Use Case code.

public interface InputMonitorTask<T extends InputEntity> extends MonitorTask {

    @Override
    default Class<? extends OdfTaskRunner<?>> getRunnerClass() {
        return InputMonitorTaskRunner.class;
    }

    Collection<T> queryInputEntities();

    void saveInputEntity(T input);

}

As you can see, there are two methods to implement:

  • queryInputEntities() contains logic to retrieve data from an external source.
  • saveInputEntity() exists because ODF 2 doesn't know what InputEntity implementation is used inside a Use Case and therefore has no knowledge of which repository to use.

The interface implementation can look like this:

@BotTask
public class EmailMonitorTask implements InputMonitorTask<Email> {
    
    private final EmailRepository emailRepository;
    private final AttachmentRepository attachmentRepository;
    private final EmailService emailService;

    @Inject
    public EmailMonitorTask(EmailRepository emailRepository, AttachmentRepository attachmentRepository, EmailService emailService) {
        this.emailRepository = emailRepository;
        this.attachmentRepository = attachmentRepository;
        this.emailService = emailService;
    }

    @Override
    public Collection<Email> queryInputEntities() {
        return emailService.readEmails();
    }

    @Override
    public void saveInputEntity(Email email) {
        emailRepository.create(email);
        attachmentRepository.createAll(email.getAttachments());
    }
}

The actual data retrieval logic is hidden inside some Use Case-specific EmailService because this isn't a point. The point is that for each returned Email, a new Transaction is started by InputMonitorTaskRunner. And when it is persisted, the runner calls saveInputEntity() to persist the corresponding Email.

Scheduling Business Process execution

With Monitor Tasks available in the Use Case developer's toolbox, it is logical to keep Business Processes running for a long time. You can either start a BP manually and stop it manually. However, instead of manual management, a good practice is scheduling BP execution at regular intervals and configuring monitors accordingly. For example, if you want to run your BP 24/7 and the BP can run for 12 hours non-stop, schedule the BP to run each 12 hours, and set up your monitors to run no longer than 11 hours and 59 minutes.

tip

For more details on Business Process scheduling, refer to the WorkFusion documentation.

If you have several Business Processes working together, like in the case of default implementation, stagger their schedules so that monitors don't run simultaneously. For example, if you have four default BPs and decide to run monitors once in 10 minutes, arrange schedules so that Intake BP starts at, for example, 12:00:00, Processing BP—at 12:02:30, Submission BP—at 12:05:00, and Error Handling BP—at 12:07:30.