Skip to main content

AGS, AMS, AMMS services down

In AutoML, there are a lot of services and components:

  • Automl-gateway-service (AGS)

  • Automl-model-service (AMS)

  • Automl-model-management-service (AMM)

Also, task-dispatcher-service (TDS) and worker-management-service (WMS) play a part, but they are mostly related to the entire BEP cluster, not only the machine learning scope. They communicate with each other and allow you to use all the benefits of WorkFusion AutoML.

Symptoms​

To find out which service is causing you trouble and whether your problem is even service-related, study the symptoms described below.

Automl-gateway-service outage​

Automl-gateway-service is used only during training or if you intentionally call REST API. Otherwise, AutoML Bridge is employed, therefore, automl-gateway-service is not used by default.

If there's a problem with the AGS service, in Control Tower event logs, you can find the following error messages:

  • Caused by: java.lang.RuntimeException: AutoML services unavailable. Response = {"body":"(http status: 503) Service Unavailable","status":"503"}

  • Start mesos job. Result:{"body":"(http status: 503) Service Unavailable","status":"503"}

Automl-model-service outage​

If the trouble is with the AMS service, in Control Tower, the execution gets stuck on the Execute AutoML Model: Prepare Input step. In event logs or Kibana, there are no error messages.

You can find the evidence of the error in the RabbitMQ queues. In particular, the Bridge queue can contain stuck messages since the Bridge service is responsible for listening to this queue and taking messages for processing. Once automl-model-service is down, these messages get stuck there.

Automl-model-management-service outage​

When the problem is with the AMM service, in Control Tower, you are immediately moved to the Manual Task step. In Marathon, you will see nothing related to ML Workers started. The event logs do not contain any error messages.

In the Results table, the _sys_automation_extract_status column should state failure. Also, _sys_automation_extract_status_reason should contain the following error message: Extract processed with correlationId: '3a3e44be-ef15-4f01-8b3b-0982a5ce838b'. Model execute failure reason: com.google.common.util.concurrent.UncheckedExecutionException Message can't be forwarded through bridge com.google.common.util.concurrent.UncheckedExecutionException: com.workfusion.spa.automl.modelmanagement.client.exception.ModelManagementClientException: <html><body><h1>503 Service Unavailable</h1>No server is available to handle this request.</body></html>.

AutoML workflow​

Be sure to understand the AutoML workflow and the interaction of the services with each other. This will make it easier to troubleshoot if the logs are unavailable.

Model execution​

During model execution, automl-gateway-service is used only if you make REST API calls to start the model execution from code. Otherwise, do not consider it during your investigation.

Check to see if automl-model-service is running. It contains many internal services that handle input processing, model preparation, artifacts downloading, and further routing of requests.

Before you start manipulating the model preparation, automl-model-service requests automl-model-management-service to check if the model exists.

After that, task-dispatcher-service moves the task from the task queue to the Worker's one in RabbitMQ, and worker-management-service starts its work, though the latter stage is out of the guide's scope.

Model training​

During training, automl-gateway-service is used. So, it is the major actor at this stage.

Then, the action moves to automl-model-service that handles the training process. It sends the task to queues, returns the training status, and manipulates model artifacts.

In the end, worker-management-service syncs the model results.

Troubleshooting​

To know how to troubleshoot, you should understand the high-level component communication specifics. The exact investigation steps are as follows:

  1. Understand the issue symptoms and review event logs and Business Process behavior.
  2. Check the service statuses from the server side using wfmanager status all. For details, refer to the Manage services guide.
  3. Focus on analyzing logs. You can search in Kibana by specific keywords from the Symptoms section. Also, refer to the Search specific message in logs topic. Alternatively, in INSTALL_DIR/supervisord/log/<app-name>.log, review the service logs one by one, based on the flow described above.

View also: