Skip to main content

Issues related to training BP failure

tip

The guide is valid for versions up to v10.2.8. From 10.2.9, the Cognitive Bot Training Business Process is obsolete.

In the WFML_Job_Data Data Store, find your training Business Process and proceed depending on your case.

Case 1: training BP failed with Terminated status​

Investigation steps​

Check how much time the model training took. If it took 12 hours plus 5 min to 30 min, it was terminated because the 12 h default limit expired.

Output​

If you confirm the issue, increase the limit in ZooKeeper. For that, update or create the /config/automl-model-service/automl.service.model.training.timeout parameter and restart automl-model-service. Otherwise, continue the investigation.

Case 2: training BP failed with Error status​

Investigation steps​

The investigation steps apply to the case when an error occurs during training:

  • Check training logs for errors. You can find the logs in the /opt/workfusion/vds-data/workers folder on Mesos or on BEP Master servers. Search by the Worker name, for instance, /opt/workfusion/vds-data/workers/log/worker/automl-model-service/com-workfusion-ml/vds-hypermodel-app/10-1-0-5/training/default/.
  • Check on Mesos UI for failed AutoML jobs with the corresponding model name.
  • To get more information in Work.AI versions, open the Cognitive Bot Training Business Process and review the event logs.

Output​

The output depends on the error message in the previous step. It can be an out-of-memory error, out-of-bound array index exception for post-processors, as well as server-side, restriction, connections errors, and so on. If you failed to resolve the issue on your own, collect logs and submit a support request via Workfusion Service Desk, adding the issue description.

Case 3: In Progress status persisting for training BP​

Investigation steps​

Such an issue can be due to something unexpected happening at the preparation step, such as the crushed uploading of a training set. Thus, the problem occurs before a training request is sent, and actual training was not started on a cluster. So, you cannot see failed jobs in Mesos.

If you open the Cognitive Bot Training Business Process, you can see it stuck on the preparation step. Check the event logs for error messages, such as invalid S3 credentials as a bot tries to upload a training set.

Output​

The output depends on the error message in the previous step. The root cause can be a connection failure, invalid credentials, and so on. If you failed to resolve the issue on your own, collect logs and submit a support request via Workfusion Service Desk, adding the issue description.

Read also: