Skip to main content

Issues related to component malfunctions

The issues related to component malfunctions include the following:

  • A Mesos Agent malfunction: no NFS storage installed or available, insufficient disk space on a node.
  • Out-of-memory errors on a component, such as Control Tower (CT), Task Dispatcher Service (TDS), WMS, Workers, and so on.

If you suspect your issue is related to a component's malfunction, act as follows:

Trace errors​

Check event logs for errors, paying attention to whether the Error Mode is on. If WMS enabled the mode, it means it is not possible to start Workers due to a problem.

In the event logs for a Business Process run, you can find the entire stack trace of a failed Worker from a Mesos Agent. This can help you understand what went wrong with a particular Worker at its start. If the reason for the failure is not clear from the event log, check the STDOUT and STDERR logs from the BEP Agent server for the failed Worker.

For instance, in the case of an NFS malfunction, you can find the following messages in the event logs:

service_name worker-management-service

message [com.workfusion.spa.ct:worker-app:10.1.0.36] Couldn't extract configuration files message [com.workfusion.spa.ct:worker-app:10.1.0.36] Couldn't extract configuration files

stack_trace java.nio.file.FileSystemException: /opt/app/workfusion/vds-data/workers/app/com.workfusion.spa.ct/worker-app/10.1.0.36: Input/output error

The messages mean that the probable cause of the issue is that the worker.jar file has not been written successfully to the vds-data NFS folder. To confirm, try checking the server system log and comparing the timings.

Check Kibana logs and dashboards​

If there are no errors in event logs, check service logs for malfunctions. For that, use Kibana dashboards and search for ERROR logs.

For instance, as mentioned before, an out-of-memory error on the Worker's side leads to its malfunction, and the Mesos watcher kills the Worker. In the logs, such a case is associated with the following stack traces:

nginx
I0312 02:36:14.771061 22949 executor.cpp:959] Scheduling escalation to SIGKILL in 3secs from nowI0312
02:36:14.791435 22963 executor.cpp:1027] Command terminated with signal Terminated (pid: 22976)I0312
02:36:15.792842 22975 process.cpp:927] Stopped the socket accept loopJava HotSpot(TM) 64-Bit Server VM warning: Exception java.lang.OutOfMemoryError occurred dispatching signal SIGTERM to handler- the VM may need to be forcibly terminatedException in thread "http-nio-41687-Acceptor-0" Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "http-nio-41687-Acceptor-0"

As a result, the Business Process is not executed. It is worth mentioning that out-of-memory errors are caused mainly by custom Bot Config Bundles or the Bot Config code. However, sometimes those can be caused by the activation of a low logging level for the Worker itself or the code executed in it. To find out the actual reason, add the following parameters to the Control Tower Worker:

-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=<path_on_the_file_system>

Output​

If you confirm the issue, fix the malfunction by eliminating the root cause if it is obvious. If you cannot fix the malfunction, escalate the issue to the Support team, attaching the logs of the failing service. Otherwise, continue the investigation.

View also: