Edit

Split and join Transactions

ODF 2 allows splitting a Transaction into multiple sub-transactions so that each resulting sub-transaction contains a single data entity derived from the data stored in the original Transaction. After that, such sub-transactions are processed the same way as common Transactions. At some point, you must join them back to the original one.

Split transactions

Transaction splitting is achieved with the help of the SplitTransactionTask interface.

public interface SplitTransactionTask<T extends OdfTransactionalEntity, O extends OdfTransactionalEntity> extends OdfTask {

    @Override
    default Class<? extends OdfTaskRunner<?>> getRunnerClass() {
        return SplitTransactionRunner.class;
    }

    TransactionalEntityRepository<T> getInputEntityRepository();

    TransactionalEntityRepository<O> getOutputEntityRepository();

    Stream<O> splitEntity(T entity);

}

It assumes that there is an entity of type T that is stored in the current Transaction. During execution, SplitTransactionRunner loads and converts it to entities of type O . Implementation of splitEntity() must take the entity of type T and convert it to any number of entities of type O. For each entity of type O, a new sub-transaction is created.

For example, let's assume that our Task takes emails and sends attached images to OCR. There are the Email and Attachment entities to store the data. To take advantage of multiple workers in the environment, you can process Attachments in parallel.

@DatabaseTable(tableName = "email")
public class Email extends InputEntity {
    @ForeignCollectionField
    public Collection<Attachment> attachments;
}

@DatabaseTable(tableName = "attachment")
public class Attachment extends DocumentEntity {
    public String url;     // will contain URL of attached image
    public String content; // will be set after OCR is done
}

Attachment is DocumentEntity and cannot be tied directly to the Transaction. Thus, you need to create a new entity called, for example, DocumentForOcr.

@DatabaseTable(tableName = "document_for_ocr")
public class DocumentForOcr extends InputEntity {
    public String url;
    public String ocrResult;
}

For brevity, we omit the creation of a repository class for this entity.

Your task for splitting the data looks like this:

@BotTask
public class SplitAttachmentsTask implements SplitTransactionTask<Email, DocumentForOcr> {

    private final EmailRepository emailRepository;
    private final DocumentForOcrRepository documentForOcrRepository

    @Inject
    public SplitTransactionProcessorTask(EmailRepository emailRepository, DocumentForOcrRepository documentForOcrRepository) {
        this.emailRepository = emailRepository;
        this.documentForOcrRepository = documentForOcrRepository;
    }

    @Override
    public TransactionalEntityRepository<Email> getInputEntityRepository() {
        return emailRepository;
    }

    @Override
    public TransactionalEntityRepository<DocumentForOcr> getOutputEntityRepository() {
        return documentForOcrRepository; 
    }

    @Override
    public Stream<DocumentForOcr> splitEntity(Email email) {
        return email.getAttachments().stream()
                .map(attachment -> {
                    DocumentForOcr documentForOcr = new DocumentForOcr();
                    documentForOcr.url = attachment.url;
                    return documentForOcr;
                });
    }
}

As you can see, each Email can have any number of Attachments, including zero. You convert Attachments to DocumentForOcr instances one to one, but you can do it another way, depending on your data model. For example, you can filter Attachments by the MIME type and skip those that are not images.

Here, you can see how the data looks before and after SplitAttachementTask is applied. Each DocumentForOcr is tied to its own Transaction. These sub-transactions retain a connection to their original Transaction called a Parent Transaction and have the same status. The Parent Transaction is marked as split and not processed further.

Join transactions

After OCR is done, you need to join all the documents back together. Let's have a new entity that represents an already OCRed email.

@DatabaseTable(tableName = "processed_email")
public class ProcessedEmail extends InputEntity {
    public Collection<String> attachments;
}

Now, implement JoinByParentTransactionTask.

@BotTask
public class JoinAttachmentsTask implements JoinByParentTransactionTask<DocumentToOcr, ProcessedEmail> {

    private final DocumentToOcrRepository documentToOcrRepository;
    private final ProcessedEmailRepository processedEmailRepository;

    @Inject
    public JoinByParentTransactionProcessorTask(DocumentToOcrRepository documentToOcrRepository, ProcessedEmailRepository processedEmailRepository) {
        this.documentToOcrRepository = documentToOcrRepository;
        this.processedEmailRepository = processedEmailRepository;
    }

    @Override
    public TransactionalEntityRepository<DocumentToOcrRepository> getInputEntityRepository() {
        return documentToOcrRepository;
    }

    @Override
    public TransactionalEntityRepository<ProcessedEmailRepository> getOutputEntityRepository() {
        return processedEmailRepository;
    }

    @Override
    public ProcessedEmail transformEntities(Collection<DocumentToOcr> entities) {
        final ProcessedEmail result = new ProcessedEmail();

        result.attachments = entities.stream()
                .map(documentToOcr -> documentToOcr.ocrResult)
                .collect(Collectors.toList());

        return result;
    }
}

After all sub-transactions reach this task, the transformEntities() method is called with all DocumentToOcr instances as an input. The output of this method is stored in the database and tied to the original Transaction. Sub-transactions are marked as joined and aren't processed further. The original Transaction continues from this place.

important

Currently, there is a fundamental technical limitation of the split and join mechanism. The data model allows binding multiple entities to a single transaction. The split mechanism handles it without any issue: in the example above, Transaction 1 can have more than one Email bound to it.

However, the join mechanism can't handle such situations and tries to produce a single entity as a result. In the example, all DocumentToOcr instances are joined into a single Email.

caution

If any sub-transactions encounter an error, it is propagated to the Parent Transaction and all of its sibling sub-transactions.