Skip to main content

Issues related to Eval job failure

An Eval job finished with the FAILED status because the Marathon job was killed or there is no process or process.error file. To investigate, act as follows:

Check training Data Store​

Check the model training Data Store. Go to Advanced > Data Stores > WFML_Job_Data. Find your training and check its status:

  • If the training was completed successfully, it has the COMPLETED status.
  • If the training was stopped due to a failure in its course, it has the FAILED status.
  • If the training was ended by a timeout (12h by default), it has the TERMINATED status.

View logs​

In Mesos and on the BEP Master server, check the event and training logs. Based on the information in the Mesos log, you can detect one of the following errors:

Total training amount limit exceeded​

Error code: Start mesos job. Result: {"body":"[7a54d693-9193-4d94-b5ee-003e766b13a4] (http status: 500) Total training amount limit exceeded. Count of trainings is 1 , limit is 1. Active trainings: [classification_9a83d372-52e6-416f-9e0a-800c695090ad]","status":"500"}`.

The error can be caused by interuption of one of the trainings by mistake, while the information about the training in progress still exists in model-service.

To resolve the issue, follow the steps:

  1. Connect to the BEP Master server via SSH and open the opt/workfusion/-automl-model/service/data directory. automl-model-service keeps information about the training process.
  2. Stop the automl-model-service.
  3. Clean up the in-progress directory.
  4. Start automl-model-service.

Java exception​

The root cause for such an exception is a code implementation issue.

Unavailability of resources for training​

Recreate Manual Task​

Try recreating the Manual Task and restart the training.

Output​

If you failed to confirm the issue, continue the investigation.

Read also: