10.1.4.1

Edit

Issues related to AutoML performance

Possible root causes for AutoML performance issues include changes in input data or utilization of cluster resources. If you suspect an issue at the AutoML step, act as follows:

Check input documents

The investigation should include the following steps:

  1. Check if the average size of documents has changed since the last execution. Also, make sure that the set of records sent by the user does not include a huge document. Try to find documents with a long processing time.

  2. Check if the number of processed documents has increased.

  3. Check the structure of XML and HTML documents. The structure is not visible during preview, but it can be very complicated and include multiple empty tags, thus causing slowdowns.

Audit resource utilization

In the Mesos UI, check cluster utilization—how many resources are available and allocated. Verify how many resources you have at the moment when you launch extraction.

In the Marathon UI, check the following:

  • The number of AutoML tasks added to RabbitMQ
  • The number of ML models running
  • The amount of resources allocated to running Workers

Check models

The investigation must cover the following:

  1. Check how many AutoML models need to be executed at the same time.

  2. Make sure no model is being trained when issues related to the Business Process performance occur.

Check Agents

In the Mesos UI, go to the Agents section. If a cluster has one Agent that never gets any jobs, whereas other Agents are loaded, this can lead to problems with installing the shared directory. In this case, try rebooting the Agent or mount the vds-data folder.

Analyze lost or failed jobs

In Mesos UI, on the main page, check the left navigator to see the number of failed or lost jobs. If the jobs are numerous, find them on the main page and check respective logs.

Check network

The increase in processing time can be due not to slow execution but rather to slow Worker preparation. To start a Worker, Worker Management Service (WMS) has to download a JAR file. If the network connection is slow, the downloading can take a lot of time. So, check the network connection and throughput.

Output

If you confirm any of the issues, consider increasing cluster resources or escalate the problem to the Support team.

View also:

← PreviousML model giving different results in two environments →