Issues related to execution failures on the Worker side
Code-related exceptions
For instance, these can be the following exceptions:
MultipleCompilationErrorsExceptionMissingPropertyExceptionjava.time.format.DateTimeParseException
If the issue comes from the <script> block (for instance, org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed:Script2_genScript_30.groovy: 26: unable to resolve class), refer to the source code of the step.
Output
If you confirm the cause of your Business Process (BP) issue is a code exception, do as follows:
- Identify the failed code line.
- Gather a log with the entire stack trace of the issue.
- Reproduce the issue in a local environment.
- Investigate the root cause based on the following instruction.
Otherwise, continue the investigation.
Out-of-memory errors
For example, these can be the following exceptions:
java.lang.OutOfMemoryError: Metaspacejava.lang.OutOfMemoryError: Java heap space
Check exception
Check if your exception includes the following line:
Terminating due to java.lang.OutOfMemoryError: Metaspace / Terminating due to java.lang.OutOfMemoryError: Requested array size exceeds VM limit / etc.
If it does, it means the error mode is enabled, and the logs of failed Workers are forwarded to the BP event logs.
Check event log
Check the event log for the Exceeded task processing retry count exception. If there is one, it means the task execution was retried three times, which is the default limit.
Starting from 10.1.4, -XX:+ExitOnOutOfMemoryError was added to Workers' JVM parameters. Therefore, if a Worker lacks memory to process a task due to a leak in Metaspace or exceeded Bot Task config defaults, it stops immediately. In this case, in the Worker's stdout log, you can see the following message (the same as in Step 1):
Terminating due to java.lang.OutOfMemoryError: Metaspace / Terminating due to java.lang.OutOfMemoryError: Requested array size exceeds VM limit / etc.
Output
If you confirm the issue is due to an OOM error, do as follows:
Gather the entire log of the failed Worker.
In
{install_dir}/vds-data/workers/app/com.workfusion.spa.ct/worker-app/{VERSION}/worker.yml, check the-XX:MaxMetaspaceSizeand-Xmxvalues.To enable heap dump generation automatically for any subsequent failures, add the following options to
worker.yml:-XX:+HeapDumpOnOutOfMemoryError-XX:HeapDumpPath=../heapdump.hprofWith a heap dump, you can investigate your OOM failure in any Heap Dump Analyzer.
Otherwise, continue the investigation.
Plugin execution failures
These can be exceptions related to pool, datastore, or release plugins. To investigate, check the event logs and workfusion.out.log.
For instance, there are known exceptions related to the pool plugin execution:
com.hazelcast.client.HazelcastClientNotActiveException: Partition does not have an owner. partitionId: 74The exception means that the Worker loses connection to the Hazelcast server, fails to reconnect, and becomes inactive.
org.webharvest.exception.PluginException: Exceeded waiting threshold for pool poolEmailIngestionThis means that the pool plugin timeout was exceeded. To increase the timeout value, refer to the guide.
Output
If you confirm the issue, gather event logs and worker logs. To identify the root cause, you can obtain the logs from Kibana or from the BEP Agent: <installation_folder>/vds-data/workers/log/worker/<control_tower_host>/com-workfusion-spa-ct/worker-app/<worker_version>/default/default.
Otherwise, continue the investigation.
Lost connection to services due to network issues
For example, this can be an issue caused by java.sql.SQLException: Connection is closed. To investigate, perform the steps below.
Check Worker log
Check the Worker's log. If you see exceptions related to connection losses, check the service or the database that the Worker tries to connect to. It should be up.
Gather logs
Gather the logs of the failing service.
Output
If you confirm the issue, hand over the gathered logs to OPS engineers. Otherwise, continue the investigation.
Resource unavailability
For example, you can get the FileNotFoundException message.
To understand which part of the code fails, gather the entire stack trace of the failed Worker from the stdout and stderror and the Worker's log file.
Output
After you gather the entire stack trace of the failed Worker, identify the failed code line and try reproducing the issue in a local environment. Otherwise, continue the investigation.
Read also: