Skip to main content

Issues related to execution failures on the Worker side

For instance, these can be the following exceptions:

  • MultipleCompilationErrorsException
  • MissingPropertyException
  • java.time.format.DateTimeParseException

If the issue comes from the <script> block (for instance, org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed:Script2_genScript_30.groovy: 26: unable to resolve class), refer to the source code of the step.

Output​

If you confirm the cause of your Business Process (BP) issue is a code exception, do as follows:

  • Identify the failed code line.
  • Gather a log with the entire stack trace of the issue.
  • Reproduce the issue in a local environment.
  • Study other troubleshooting instructions on the Investigate non-operational Business Process step.

Otherwise, continue the investigation.

Out-of-memory errors​

For example, these can be the following exceptions:

  • java.lang.OutOfMemoryError: Metaspace
  • java.lang.OutOfMemoryError: Java heap space

Check exception​

Check if your exception includes the following line: Terminating due to java.lang.OutOfMemoryError: Metaspace / Terminating due to java.lang.OutOfMemoryError: Requested array size exceeds VM limit / etc.

If it does, it means the error mode is enabled, and the logs of failed Workers are forwarded to the BP event logs.

Check event log​

Check the event log for the Exceeded task processing retry count exception. If there is one, it means the task execution was retried three times, which is the default limit.

Starting from 10.1.4, -XX:+ExitOnOutOfMemoryError was added to Workers' JVM parameters. Therefore, if a Worker lacks memory to process a task due to a leak in Metaspace or exceeded Bot Task config defaults, it stops immediately. In this case, in the Worker's stdout log, you can see the following message (the same as in Step 1):

Terminating due to java.lang.OutOfMemoryError: Metaspace / Terminating due to java.lang.OutOfMemoryError: Requested array size exceeds VM limit / etc.

Output​

If you confirm the issue is due to an OOM error, do as follows:

  • Gather the entire log of the failed Worker.

  • In {install_dir}/vds-data/workers/app/com.workfusion.spa.ct/worker-app/{VERSION}/worker.yml, check the -XX:MaxMetaspaceSize and -Xmx values.

  • To enable heap dump generation automatically for any subsequent failures, add the following options to worker.yml :

    -XX:+HeapDumpOnOutOfMemoryError

    -XX:HeapDumpPath=../heapdump.hprof

    With a heap dump, you can investigate your OOM failure in any Heap Dump Analyzer.

Otherwise, continue the investigation.

Plugin execution failures​

These can be exceptions related to pool, datastore, or release plugins. To investigate, check the event logs and workfusion.out.log.

For instance, there are known exceptions related to the pool plugin execution:

  • com.hazelcast.client.HazelcastClientNotActiveException: Partition does not have an owner. partitionId: 74

    The exception means that the Worker loses connection to the Hazelcast server, fails to reconnect, and becomes inactive.

  • org.webharvest.exception.PluginException: Exceeded waiting threshold for pool poolEmailIngestion

    This means that the pool plugin timeout was exceeded. To increase the timeout value, refer to the guide.

Output​

If you confirm the issue, gather event logs and worker logs. To identify the root cause, you can obtain the logs from Kibana or from the BEP Agent: <installation_folder>/vds-data/workers/log/worker/<control_tower_host>/com-workfusion-spa-ct/worker-app/<worker_version>/default/default.

Otherwise, continue the investigation.

Lost connection to services due to network issues​

For example, this can be an issue caused by java.sql.SQLException: Connection is closed. To investigate, perform the steps below.

Check Worker log​

Check the Worker's log. If you see exceptions related to connection losses, check the service or the database that the Worker tries to connect to. It should be up.

Gather logs​

Gather the logs of the failing service.

Output​

If you confirm the issue, hand over the gathered logs to OPS engineers. Otherwise, continue the investigation.

Resource unavailability​

For example, you can get the FileNotFoundException message.

To understand which part of the code fails, gather the entire stack trace of the failed Worker from the stdout and stderror and the Worker's log file.

Output​

After you gather the entire stack trace of the failed Worker, identify the failed code line and try reproducing the issue in a local environment. Otherwise, continue the investigation.

Read also: