10.2.7

Edit

Issues related to Eval job failure

An Eval job finished with the FAILED status because the Marathon job was killed or there is no process or process.error file. To investigate, act as follows:

Check training Data Store

Check the model training Data Store. Go to Advanced > Data Stores > WFML_Job_Data. Find your training and check its status:

  • If the training was completed successfully, it has the COMPLETED status.
  • If the training was stopped due to a failure in its course, it has the FAILED status.
  • If the training was ended by a timeout (12h by default), it has the TERMINATED status.

View logs

In Mesos and on the BEP Master server, check the event and training logs. Based on the information in the Mesos log, you can detect one of the following errors:

Total training amount limit exceeded

Error code: Start mesos job. Result: {"body":"[7a54d693-9193-4d94-b5ee-003e766b13a4] (http status: 500) Total training amount limit exceeded. Count of trainings is 1 , limit is 1. Active trainings: [classification_9a83d372-52e6-416f-9e0a-800c695090ad]","status":"500"}`.

The error can be caused by interuption of one of the trainings by mistake, while the information about the training in progress still exists in model-service.

To resolve the issue, follow the steps:

  1. Connect to the BEP Master server via SSH, and open the directory: opt/workfusion/-automl-model/service/data. automl-model-service keeps information about all training process.
  2. Stop the automl-model-service.
  3. Clean up the in-progress directory.
  4. Start automl-model-service.

Java exception

The root cause for such an exception is a code implementation issue.

Unavailability of resources for training

Recreate Manual Task

Try recreating the Manual Task, and restart the training.

Output

If you failed to confirm the issue, continue the investigation.

Read also:

← Issues related to training BP failureIssues related to bean creation error →