Apply BEP practices

With the introduction of the BEP execution platform:

• Bot config tasks are not executed inside Control Tower (CT) JVM but inside separate JVMs running on different machines.
• You may execute tasks in different JVMs on different machines even from the same Business Process (BP) or the same BP step. Executing two tasks inside the same BP is a coincidence—the larger cluster, the more worker processes are running, and the less likely tasks will be processed by the same worker.
• A worker may crash or be killed during the task processing. In this case, the task is processed by another worker on the same or a different machine from the beginning.
• For versions below 10.0, all non-RPA CT tasks are executed inside CT JVM, which has large enough RAM (16 GB) shared between CT and bot executions. With BEP, each worker usually has much less memory than available for the CT process. For example, CT worker version 10.2 default memory settings: 1 GB heap + 512 MB metaspace.
• Some plugins, for example, cache or pool, now use network communication or interaction with the Hazelcast service.
• Most plugins that require interaction with CT use REST; while part of them, like some datastore plugins, apply direct connections to the target system, for example, DBMS.

Distributed execution

Don't access CT components directly

As a bot is executed in a separate worker process on a different node, you can't directly access any CT components.

The code below doesn't work:

// this code does not work!
service = ApplicationContextProvider.getContext().getBean(SomeCtService.class);
service.doSomething();

Don't call AutoML services directly

Don't use:

• Deprecated automation—* set of plugins.
• Direct access to AutoML REST services by creating custom clients (HttpClient, Feign, RestTemplate, and so on) or using the http-extended plugin.

Use special Model steps. For more details, refer to Add and configure Model and Dataset steps in Business Process.

Don't use in-memory mechanisms to pass data between steps

Don't use singletons or static variables to pass data between steps. You can execute different steps in different JVMs on different nodes, so setting values to variables (event static) won't be visible in other JVMs.

To pass data between steps, use ODF transactions, Data Stores, S3 storage, and so on.

Don't use local filesystem to pass data between steps

You can execute different steps in different JVM on different nodes. Thus, writing a file in one step may not be available to another step if a task is executed on a different node.

caution

In the development environment or single-node clusters, passing data through a local file may work correctly, but it will fail on the multi-node cluster in the production environment.

To pass data between steps, use ODF transactions, Data Stores, S3 storage, and so on.

Don't pass extensive data between steps as step output

Don't put extensive data into setup output using the export plugin. This leads to the increased network load with each result passed by the network as messages from a worker to RabbitMQ, and then to CT. The result is also the excessive size growth of CT tables, which, in turn, causes a slowdown in task processing in CT.

To pass data between steps, use ODF transactions, Data Stores, S3 storage, and so on.

Perform interrupted and continued task execution

Each task can be interrupted at any moment—a worker may crash, be killed, and so on. Such tasks are returned to the worker queue automatically and consumed by another worker. In this case, the task is reprocessed from the beginning.

For a long-running task, it may lead to repeating the long-executing part of the task. To avoid this, consider splitting one big task into smaller ones.

If some part of the task is sensitive to repeatable execution, for example, money transfer, add a mechanism that prevents repeating a sensitive operation. In the case of DB operations, it may look as follows (all the data is stored in the DB):

// note1: this is just an example, actual implementation must be implemented regarding your storage mechanism (DBMS, etc.)
// note2: exceptions handling is omitted in this piece of code
if (getOperationStatus('operationA') != COMPLETED) {
    beginTransaction();
    doOperationA(); // main phase
    setOperationStatus('operationA', completed)
    commitTransacation();
}

The recommendations are as follows:

• Split large, long-running tasks into smaller ones to avoid repeating a large amount of long-running logic in case of a worker crash or kill.
• Make task implementation idempotent, use checkpoint saving or checking, and so on.

Configure waiting for condition—use release plugin

Sometimes, a step should wait for some condition in the external system. To check the condition, perform a periodical call to an external service.

Don't use a cycle with Thread.sleep() inside, as this leads to a situation when a worker is busy by the "waiting" state, thus under-using cluster resources.

In this case, it is recommended to use the release plugin. If the condition is not reached, exit by the release plugin. A task is returned to CT and re-submitted again to BEP in one minute. This approach is better as after the release worker finishes processing the task and can process other tasks.

Apply Data Store plugin with direct DB connection

Data Store plugins use a direct JDBC connection for queries. This may cause a problem, for example, in RPA tasks when RPA nodes have no access to the database for security reasons.

Ensure that all cluster nodes access the database (a process can open DBC connections) in the customer environment. If, for example, RPA nodes have no access (cannot open connections) to DBMS, avoid using the datastore plugin in the RPA steps.

Control resource usage

Check memory consumption

When a task needs to process some document, pay attention to the following points:

• The library used for processing the document can load the whole document into memory.
• The actual production document set may contain much larger documents than an average document or documents used during the step development.

The recommendations are as follows:

• Test your bot config with large real or generated documents. Try to get information from a customer about average and maximum document sizes. Run a bot step in JVM with the same memory settings as in the production environment. If you don't know the actual memory setting for production, use the default value defined in worker.yml inside the worker app for a given version.
• Monitor memory consumption during bot step execution with a large document. For memory monitoring, you can use Azul JDK, VisualVM, or any other tool of your choice.
• Use stream-based frameworks or stream-based modes for frameworks that work with the document.

To work with large files, mind the following recommendations:

• XML: Use SAX or StAX parsers instead of DOM.
• MS Office Documents using POI: If possible, use SXSSF API. It does not support the document modification.
• PDF: Use PDFBox.

Apply cache and pool plugin—don't store large objects

Don't store large or potentially large objects in the cache or pool plugins.

The cache and pool plugins store and share their data using the Hazelcast server. Storing large objects may have a negative impact on the network traffic, memory consumption, and even produce OOM errors in the Hazelcast server app.

Use only simple value objects (String, Integer, and so on) in the cache and pool plugins when possible. Don't use large or potentially large objects in these plugins.

Avoid creating big amount of threads, check threads closing

Don't create any other thread in the task execution code unless needed. Note that a single node may run multiple workers (tens of workers). Each thread in Java uses a kernel thread. Thus, if each worker creates ten threads, hundreds of them are created. This leads to unnecessary context switching unless the node's CPU has hundreds of cores, which is unlikely in most cases, and also may lead to the kernel thread limit exceeding the situation.

Make sure any created and started thread is stopped and destroyed correctly, even if task processing fails. Otherwise, the worker process accumulates more and more useless threads consuming resources (memory, kernel threads, and so on).

Some frameworks may start background threads during initialization, opening a connection, and so on. If a framework doesn't close or finalize correctly after task processing is completed, these threads retain running.

If such framework is a part of BCB, it loads in a separate class loader for each new task and won't reuse the already started threads but create new ones. After multiple steps, a single worker process may result in a considerable amount of useless threads consuming resources.

The recommendations are as follows:

• Avoid creating new threads when possible. Ensure all the created threads are stopped and destroyed for successful task completion and errors or exceptions.
• Check used frameworks, especially if such framework is a part of BCB for threads it started in the background. Ensure that all background threads are stopped correctly both for successful task completion and errors or exceptions.

Ensure streams closing

If a step opens any kind of a stream (file, network, and so on), make sure that all of them are closed correctly. Always close streams, files, and so on, don't rely on the stream auto-close on GC. Check for possible connections opened in the background by frameworks. Make sure such connections are closed after task processing is completed.

Note that some framework opens connections in the background, and you must ensure that such connections are closed after the task is completed. Don't try to re-use connections between tasks because:

• They may run on different nodes.
• Each task loads BCB classes in a separate class loader. Thus, in most cases, new connections or streams are created.
• You don't know when the stream or connections should be finally closed.

tip

For Groovy scripts, see Working with IO.

Don't use share static GsonUtils in bot step code

Some of the shared libraries provide a static helper class that is GsonUtils with preconfigured Gson serializers: GsonUtils.GSON, GsonUtils.GSON_PRETTY, and so on.

Don't use them in bot configs, as each time you serialize or deserialize some classes from BCB, it keeps a reference to the (de)serialized class, thus preventing it from garbage collecting. As you use a new class loader for each task, even the same class (a class with the same FQN) is considered a different class for the next task and thus stored and not GC-ed. Eventually, it leads to an OOM Metaspace error.

Don't use GsonUtils from shared libraries:

• com.workfusion.common.utils.GsonUtils
• com.workfusion.utils.gson.GsonUtils
• Any other

Don't use static variables from shared libs in bot step code

Some of the shared libraries provide static mutable variables and methods with state, for example, javax.mail.Session#defaultSession and the related javax.mail.Session#getDefaultInstance method that can be initiated only once.

If different tasks from the same or different BPs try to use such a feature, it can lead to unexpected behavior, for example, the variable value doesn't equal the expected.

The recommendations are as follows:

• Don't use static variables, static methods with the state from the shared library.
• If you need it, you must control the value of the method field or state.

Justify usage of multi-process ODF 2 design

Apply all solution logic that fits single BP

All logic is implemented in one BP presenting:

• A simple e2e flow.
• No complicated exception handling with automatic reprocessing from the middle.

Apply cleaner solution logic design using multiple BPs

Split the solution logic between multiple BPs where a Transaction goes sequentially from a BP to a BP. Create the logic to start processing a Transaction in the next BP after the previous BP has finished execution.

The best usage cases are as follows:

• A separate Intake BP helps isolate the main processing logic from the external system and reprocess the same data multiple times without changes in the external system.

• A separate Submission BP is needed when you have the exception handling logic that can reprocess only this Submission BP if it has some problem during the first execution of this BP. Also, you can create this BP for reusability or if you have complicated logic inside, for example, multiple systems with complex rules.

• A separate ExceptionHandling BP is helpful when you wish to have a human-in-the-loop for errors review or if you have specific logic to reprocess execution of the Transaction in the BP where it failed before.

The flexible ODF 2 multi-process design allows extending the solution with any additional BPs. Such custom BPs can be helpful when you have the logic to work with aggregated data, need some utility logic (Data Purge or health check), or manipulate with a different group of entities, for example, to avoid the complicated split-and-join logic.

Continuous data fetch design

Keep in mind potential issues for continuous data fetch:

• Double processing. The next read iteration can happen when the previous one is not finished yet. If it can be a problem from the business point of view, you need to implement the logic that can prevent it:

  • Use a Data Store to set meta-information (status of all iterations or last read ID) and use this info in each new iteration.

  • Implement the logic in the BP to skip duplicates (by external ID, hash of content).

• Data loss. You can read data and mark it as in progress but fail in CT, for example, when completing a bot step. As a result, this batch of data will be lost. Options to mitigate it are as follows:

  • Split the logic to read data into two steps: the first bot step reads data portion from the external system and splits it by Transactions, and the second one marks each Transaction as in progress in the external system.

  • Add some logic to check that all existing Transactions in a Data Store has a proper status (for example, in progress if this status set at the end of loop logic, which means that this Transaction handled by your BP after reading of it) and try to reprocess it again (read and add to the result list or create a new Transaction for original documents that your BP did not handle).

info

Use a schedule or trigger a BP using WorkFusion API only if you need to launch your BP a few times per day or even less often (once per week, once per month). Use an infinite monitor loop for all frequent cases.

Configure infinite monitoring loop in one BP

Create a BP where one bot reads data from the external system and adds one extra record for the result collection. Then, go to the rule in the BP and push forward all Transactions and artificially-generated additional records back to read the data step.

If there is no new data, use the release plugin (or RetryRequiredExceptionin ODF 2) to execute this step again after some delay. Delays are configurable per each Monitor class.

The pros are as follows:

• As there is only one active BP, it is easy to find and troubleshoot. CT has fewer BP runs in the system database, making the CT UI faster.

• Do not generate useless data in the CT DB for a BP if there is no new data in the external system.

• The flexible interval configuration means you can make a delay between read dynamic based on the total load.

• A soft stop is available in ODF 2. To avoid data loss that the CT stop action can create if some record is in progress, instead of using a stop link, you can navigate to the Monitors Data Store and manually set the stopped field to have a particular Monitor softly stop.

The cons are as follows:

• The CT Data Purge does not clean Data Store data for active BPs.

• Stop a BP manually or use a schedule if a new instance of the BP is launched. Or you must implement the additional logic in a bot step to finish BP execution if a new instance of this BP is created.

• If there is some bug in the logic and the monitoring logic is stuck, the solution is stuck until somebody escalates and fixes.

Use schedule to trigger new BP for reading

Create a BP that reads the data from the external system. Set up a schedule to run this BP every X minutes or hours.

The pros are as follows:

• It is easy to stop monitoring. Pause your schedule and wait for the current BP to finish.

• There is no need to implement additional logic to let CT Data Purge work out of the box.

• If one BP is stuck because of some issue, the next BP is launched according to the schedule. As a result, the monitoring logic does not get stuck.

• It is easier to solve a data loss issue. The first bot step reads data from the external system and splits it by Transactions, while the second bot marks each Transaction as in progress in the external system. For example, for a loop BP, an extra record for loopback can be processed faster and returned to the first read until you mark all the Transactions as in progress in the external system and it will produce duplicated Transactions in your BP, or you need to add additional logic to avoid it.

The cons are as follows:

• The approach generates many BPs and can decrease the CT UI performance.

• The approach generates a BP and uses BEP cluster resources even if there is no new data in the external system.

• It is difficult to find your BP instance when you need to view execution details.

Trigger BP using WorkFusion API or plugin

You can use the solution when you have dependent BPs and don't want to have an infinite loop in this BP.

The pros are as follows:

• BEP resources are saved, and the BP execution is started only when there is data for processing.

• Faster E2E means no need to wait for the next loop iteration.

The cons are as follows:

• The approach generates a lot of BPs and can decrease the CT UI performance.

• It is difficult to find your BP instance when you need to view execution details.

Limit parallel execution

Use Bot Sources to limit the parallel execution of some logic within a Bot Task. If you use the pool plugin, estimate the possible impact on the BEP cluster first.

Apply Bot Source approach

If you use the Bot Source approach:

• A Worker is not blocked when the limit of parallel execution is achieved.

• As you configure in the UI, you don't need to recompile the BCB.

caution

The approach impacts the E2E processing time as a message will have an extra loop in the queue.

Use pool plugin

If you use the pool plugin approach:

• It is easy to find the usage in the scope of one project (search the source code).

• There is a low impact on the E2E processing time as only part of the bot logic is blocked from the execution, and the execution continues immediately after the resource is unblocked.

caution

A worker is blocked from executing other tasks while waiting. The cluster can get stuck for a while if there are many such tasks and the critical section is not fast.

You also need to have a strong naming convention to avoid a possible impact on other solutions.

Data manipulation

• Make ORM mapping as simple as possible. Map to another entity only if you use this mapping in your logic, and it does not bring overhead during lazy loading. Sometimes, it's better to add an extra method to the repository to load a collection by some ID.

• Do not use a local disk to share data between bot steps. BEP can execute the next step on another server.

• Store files and large text to the file storage (S3) and everything else—in Data Stores.

• Export as fewer data as possible between bot steps. It will decrease the load for Data Purge and speed up the transition between steps.

• The solution configuration must have one entry point, for example, a configuration Data Store. Do not mix multiple configuration approaches (ETL steps + Data Stores).

• Do not design a DB schema to simplify analytics. Build it to speed up the solution processing. Analytics dashboards must use Data Marts and should not depend on operational Data Stores that can have data purged.

• Think which columns in Data Stores must have indexes when designing Data Stores.

Exception handling

• Do not fail a bot in the middle of a BP. In this case, you can't automate exception handling, notifications, and reprocessing. You can do it only during a development phase to have a possibility to reprocess this step and check that your fix works.

• Catch an exception, store it into a Data Store, record into the event log, and notify SMEs if needed.

• If you skip bot step execution, the bot step will not add custom data to export to the next step. As a result, a business rule can get stuck for this record, or a Manual Task can fail on rendering.

• Design business rules to be able to move a failed Transaction into a proper flow.

• Add business rules to skip steps that don't have the "skip failed Transaction" logic (Manual Tasks, out-of-the-box non-ODF steps).

• Notify SMEs about an error and provide them with a simple way to trigger processing from scratch for the same document instead of implementing a complicated exception-handling logic.

• Create a separate Exception Handling BP if you have a complicated logic for a different case or a logic to reprocess transactions that failed not from the first step.

• If you have a reprocessing logic, skip execution for bots completed during the first run. Add the logic at the beginning of a bot step in the current task or the Runner class.

You can have a different logic to reprocess a failed BP:

• Use a Manual Task (or automatic logic in a bot step) in the Exception Handling BP to rerun your Transaction through a specific BP. In this case, update statuses so that the required BP handles this document.

• Use a Manual Task as a trigger in your BP (the first step). In this case, an SME can resubmit the same document, and you should have the logic to understand that you tried to process it before.

Health check

Validate that the production environment is properly configured and can be used as a smoke test for the solution after the environment upgrade.

You can implement the following checks inside a health check BP:

• Validate that required configuration parameters are not empty.

• Validate that you can read secrets by aliases from your configuration.

• Validate that required assets are in place, for example, template files on a local drive or S3.

• Try to connect to the external API using provided URL and credentials.

• For RPA systems, try executing a simple read action, for example, login or get a list of data.

• Add custom checks for areas that are not stable (depends on the screen resolution, a required proxy, or certificates).

• If there are many nodes for one fleet, you can have RPA checks on each node. Provide an input file with the number of records bigger than the cluster side and add some sleep in an RPA step to guarantee that it can't execute this step fast to avoid a case when not all nodes take at least one task.

The recommendations are as follows:

• Log each check result (success or failure) into the event log.

• Do not kill a bot if one check fails. Catch all errors for each check to get a full report about all issues after one BP run.

• Do not add a BP package for a health check BP into the asset bundle. You don't need to create a new BP instance after each deployment. Create a BP from bot steps when you need to launch it.

Operational Data Purge

CT purges the data in internal tables, but it doesn't recognize the user-defined Data Store structure. You are responsible for Data Stores and S3 cleanup. It can block the data intensive solution if there is no Data Purge for Data Stores or S3.

To implement the Data Purge logic:

• Define the rules on what data can be removed and when. Create a Java class for the configuration. For example, remove all completed Transactions with related data older than one month and Transactions with errors older than three months.

• Remove a Transaction object at the end of the BP run. Remove data that don't have a hidden reference to other resources first. For example, remove content from S3 before removing a record in a Data Store that has a link to this file.

• Use a bulk delete operation instead of deleting each row in a separate Transaction. You can use raw SQL queries to optimize the speed.

• Extract the logic to remove data from a Data Store into a separate Java method to simplify support and extension of the logic.

The recommendations are as follows:

• It doesn't give huge benefits for low-volume solutions, for example, if they generate data in DS less than 100 MB per month.

• Do not add a BP package for Data Purge into an asset bundle. There is no need to create a new BP instance after each deployment. Create a BP from bot steps when you need to launch it.

• Set up a schedule for this BP for every X days based on the amount of data your solution generates.

Notifications

• Implement the notification logic in one place, for example, a service or a runner, and call it from where you need it. For example, your last step in the BP can be responsible for all notifications (if you don't have specific requirements). Thus, you need to define a notification type in the previous bots or in the step based on the entity's state.

• Define scenarios for successful and error cases:

  • If you processed a Transaction successfully, send a notification or an aggregated report per day or do not send it.

  • In case of a failure you try to handle automatically, notify an SME about it.

  • If you send a task to manual handling, you need an additional notification, for example, an email.

• Send an aggregated report for multiple Transactions where possible to avoid spamming SMEs.

• Make templates and email recipients for notifications configurable during the production, for example, in a Data Store.

Performance optimization

• Use asynchronous API calls where possible. Use the release plugin or RetryRequiredException (for ODF 2) while waiting for a response. In the case of long-running synchronous APIs, you block one BEP worker while waiting for an API response.

• Always use OCR in asynchronous way: one bot sends a document to OCR, and the second one waits for a result (using the release plugin or RetryRequiredException) and saves results at the end. Otherwise, you block one BEP worker while waiting for an OCR response.

• Use Model steps to call models. Otherwise, you block one BEP worker while waiting for a model response.

• Do not use Thread.sleep() in your logic with a value bigger than 5-10 seconds. It blocks a worker from processing other tasks in the queue.

• Export as fewer data as possible between bot steps. Store documents on S3 as soon as possible and provide only a link to the document between steps (if needed).

Sometimes, you can use an RPA node for a non-RPA task. For this, you need to have the robotics plugin in your bot XML step:

• It can be helpful if you have limited resources in the BEP cluster with high utilization, but the RPA node has almost no work.

• Working with large files typically requires increased memory for worker machines to process them. You can dedicate one RPA node for such tasks and give extra resources only for this RPA worker instead of providing additional memory for all BEP workers and decreasing the amount of a BEP worker that you can launch in the cluster.

Poor solution design examples

• Don't read emails, store them in the DB, and mark them as read in one bot step. Assume the bot fails in the middle of a loop for marking an email as read. As a result, Transactions for emails that are already marked as READ don't export to the next step and won't be fetched during the next read operation.

• Don't use Thread.sleep(). Assume a developer has a bot step with the logic to sleep for five minutes (the logic of waiting for some updates). As a result, the WorkFusion support team will receive an issue ticket with a complaint that the cluster is stuck. The issue is caused by BEP workers with tasks having five-minute Thread.sleep().

• Don't use OCR in synchronous way. Assume a developer sends to OCR and waits for a result using a while loop in one bot step. As a result, the WorkFusion support team will receive an issue ticket with a complaint that the production is stuck and works very slowly. The issue is caused by the solution having a lot of tasks for OCR, so that the environment can use only two (out of dozens) available BEP workers.

• Don't export all data between steps, including output from a Manual Task after Information Extraction. Assume the out-of-the-box Data Purge cannot clean up the DB faster than a BP generates the data. In this case, a custom Data Purge needs to clean up the DB more aggressively.

• Don't use @ForeignCollectionField in ORMLite mapping. Assume ORMLite has connections leak when you iterate through this collection using streams. Finding which BP is the root cause of connections leak is a problem. The solution is to have a separate method in the repository to get this collection. This allows getting all required data in one query and will not cause any issues.

• Don't use excessive event logging. A CT worker has access to log an object, which allows sending log events to CT where they are displayed in the UI. Each event is sent through a RabbitMQ message stored in the CT database and displayed in the CT UI. Sending tons of such messages has negative consequences like network and disk load, an over-growing table in the CT database, UI slowness in CT.

  Control the number of log events sent to CT and strive to minimize the amount of such log events. These logs are mainly for a business person who uses CT, so avoid logging technical information. For a technical log, use a standard logger created by LoggerFactory—view such log in Kibana.

• Don't use undocumented features like configuration, properties, and so on, that may be subject to change in the future. When using some undocumented feature, be ready that a bot configuration may work incorrectly or fail after the platform upgrade. If you have questions, consult with the component team for the best solution or create a new feature request.