Skip to main content

All Business Processes are stuck

Possible causes of issues related to no visible progress in running Business Processes (BP) can be as follows:

  • Process code itself

  • Infrastructure problems

Faulty Business Process design​

Probable root causes related to the BP design include:

  • Improper conditions in loops, leading to infinite execution or non-limited requests for external resources and a long wait for their availability. Since all Workers are busy with such tasks, there is no visible progress for other processes.

  • Documents stuck on the intake stage, like in the example below. A document with a missed value in one of the fields causes NullPointerException, and the process tries to take it many times with no luck. Fixing it requires changing or removing faulty data.

    org.webharvest.exception.ScriptException

    Config line 4: script block
    com.workfusion.idp.common.exceptions.ProcessingException: Cannot process transaction
    at org.webharvest.runtime.scripting.GroovyScriptEngine.eval(GroovyScriptEngine.java:141)
    Caused by: com.workfusion.idp.common.exceptions.ProcessingException: Cannot process transaction
    at com.workfusion.idp.usecase.app.AppWithExceptions.processTransaction(AppWithExceptions.java:134)
    at org.codehaus.groovy.vmplugin.v7.IndyInterface.selectMethod(IndyInterface.java:237)
    ... 25 more
    Caused by: java.lang.NullPointerException
    at java.lang.String.contains(String.java:2133)
    at com.workfusion.idp.usecase.utils.HtmlUtils.pageContains(HtmlUtils.java:42)
    ... 31 more

To investigate the issues in detail, refer to Verify symptoms of non-operational Business Process.

Service failures and connectivity issues​

Worker memory issue​

The issue causes continuous termination due to java.lang.OutOfMemoryError: Java heap space.

For details, see Issues related to execution failures on Worker side.

RabbitMQ connection issues​

These issues can leave traces in the logs:

  • RabbitMQ log

    ** ERROR ** mnesia_event got {inconsistent_database, running_partitioned_network, 'rabbit@int3-workfusion.com'}

  • Task Dispatcher Service (TDS) log

    "message": "Queue not present, scheduling consumer SimpleConsumer [queue=exec.workfusion.com.task.17e2cae4-436f-4150-aa98-7b05f63fc650-4069, consumerTag=null identity=63d4af6e] for restart",
    "logger_name": "org.springframework.amqp.rabbit.listener.DirectMessageListenerContainer",
    "stack_trace": "java.io.IOException: null
    at com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:126)
    at com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:122)
    at com.rabbitmq.client.impl.AMQChannel.exnWrappingRpc(AMQChannel.java:144)
    at com.rabbitmq.client.impl.ChannelN.queueDeclarePassive(ChannelN.java:1006)
    ... 43 common frames omitted
    Caused by: com.rabbitmq.client.ShutdownSignalException: channel error; protocol method: #method<channel.close>(reply-code=404, reply-text=NOT_FOUND - no queue 'exec.workfusion.com.task.17e2cae4-436f-4150-aa98-7b05f63fc650-4069' in vhost 'bep', class-id=50, method-id=10)
    \tat com.rabbitmq.client.impl.ChannelN.asyncShutdown(ChannelN.java:516)

These messages mean that those were network connectivity issues that led to the RabbitMQ cluster inconsistency. The solution is to restart the service.

Task Dispatcher Service issues​

The following issues are possible:

  • The service configuration is incorrect.

    BP executions are stopped due to the exception in the TDS:

    com.google.common.util.concurrent.UncheckedExecutionException: java.lang.IllegalStateException: Failed to check status node for BP instance '495b3814-aca0-4e59-b3a7-959c0c88b6e0'
    at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2050)

    The described issue was caused by the error below:

    Failed to start connector [Connector[HTTP/1.1-9090]]","logger_name":"org.apache.catalina.core.StandardService","thread_name":"main","level":"ERROR","level_value":40000,"stack_trace":"org.apache.catalina.LifecycleException: Failed to start component [Connector[HTTP/1.1-9090]]
     at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:167)
     at org.apache.catalina.core.StandardService.addConnector(StandardService.java:225)

    Based on this message, the embedded Tomcat failed to start due to the port issue at the TDS startup. In this case, an infrastructure fix and service restart should help resolve the problem.

  • TDS is not subscribed to any RabbitMQ queue.

    In the TDS logs, you can see the following error:

    "message":"There is some exception on process scheduler task.",
    "logger_name":"com.workfusion.spa.core.service.task.dispatcher.configuration.TaskSchedulerErrorHandler",
    "stack_trace":"java.lang.IllegalStateException: Failed to retrieve status node for BP instance '69071138-1748-40aa-aa5e-4943441f70b0'
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      at java.lang.Thread.run(Thread.java:748)

    Check if the mentioned BP UUIDs are present on the ZooKeeper path: /control-tower/execution/active-instances.

    The solution is to remove all records from the path via the deleteall command in the ZooKeeper console: "deleteall /control-tower/execution/active-instances".

    After that, Workers should start to scale, and BPs—to move forward.

BEP Agent, BEP, and RPA Worker issues​

Other issues include those caused by hanging of particular steps, like OCR processing. This is related to the service unavailability or connection to RabbitMQ for task consumption. For detailed instructions, refer to the Troubleshoot infrastructure article.

View also: