9.2

Edit

Machine Learning support

EGID

Egid is the experiment group ID. You can find it in the WFML_Job_Data WF Data Store. Egid contains Business Process (BP) UUID. You can find it in the link to the Automation BP: /workfusion/secure/business-process/edit/0e6aab2e-bbc3-4745-b326-dc5cc8053aeb. Use the BP UUID to search for Egid.

In Marthon and Mesos, the _ (underscore) character is replaced with - (hyphen): message_0e6aab2e-bbc3-4745-b326-dc5cc8053aeb→message-0e6aab2e-bbc3-4745-b326-dc5cc8053aeb.

From WF version 8.5.1 and above, for extract jobs, the _ (underscore) character is replaced with -- (double hyphen).

wfmanager

To manage the service, use the wfmanager utility:

wfmanager (start | stop | restart | status ) <service_name>

Example:

wfmanager restart mesos-adapter

# Shows status of all services
wfmanager status

Additional information about VDS components

Investigation instruction

The investigation starts from an Automation BP. The Automation BP event log contains errors that are not related to VDS services. After that, go to the other end of the chain—Mesos, where you can see if training or extraction began.

  1. Check the Automation BP event log. If it contains an error, try to find the solution on the Automation Business Process errors page.

    For extraction, check the _sys_automation_extract_status_reason column in the table from the Results tab and the Data menu for the BP. The column contains an explanation of the extraction status. To find solutions for statuses, navigate to the Record after extract page and go to Failure outcome.

  2. Check Mesos. The default link is :5050. To see the logs of the current job, click the Sandbox link. On the opened page, check Stdout and Stderr logs. Check mesos-master.out.log (/supervisord/log/mesos-master.out.log) and mesos-master.err.log (/supervisord/log/mesos-master.err.log). You can find solutions for some errors on the Errors page.

    MESOS task states:

    • RUNNING: the task is now active.

    • KILLED: the task was completed successfully.

    • FAILED: the task was unexpectedly stopped because an error occurred.

    • LOST: an agent was disconnected from the master due to network problems.

    If an active task has a lifetime of more than 12 hours, download and analyze a thread dump for the current job. You can download it in Marathon.

    Expand to read instructions

    From Task ID, you can get the path to the application. You need the Task ID part before the dot. For example: extract_flat-prod_presentation-information-extraction_8.5.5.1_message-177a8e47-272b-4150-943a-52482dfdf21c.f8e88885-3263-11e8-ab34-fa163e1fc8a8.

    Replace _ (underscore) with / (slash). For example, extract_flat-prod_presentation-information-extraction_8.5.5.1_message-177a8e47-272b-4150-943a-52482dfdf21c gives the following path: extract/flat-prod/presentation-information-extraction/8.5.5.1/message-177a8e47-272b-4150-943a-52482dfdf21c.

    The last part of the path is the application name, for example,message-177a8e47-272b-4150-943a-52482dfdf21c.

    In this folder, for example, extract/flat-prod/presentation-information-extraction/8.5.5.1, click the application name.

    On the next page, click the link under the application ID to open the actuator.

    Download the dump and analyse it or attach to ticket for further investigation.

  3. Check Marathon. The default link is :8480. Check marathon.out.log (/supervisord/log/marathon.out.log) and marathon.err.log (/supervisord/log/marathon.err.log). You can find solutions for some errors on the Errors page.

  4. Check Mesos Adapter or Scaling service logs.

    Evaluation: check the Mesos Adapter log (/supervisord/log/vds-mesos-adapter.out.log). You can find solutions for some errors on the Mesos adapter errors page.

    Extract: check the Scaling service log (/supervisord/log/vds-scaling-service.out.log). You can find solutions for some errors on the Scaling service errors page.

  5. Check RabbitMQ. The default link is :45672. Make sure that it's available.

  6. Check the Gateway service log (/supervisord/log/vds-gateway-service.out.log. You can find solutions for some errors on the Gateway service errors page.

  7. Check the Nginx log (/supervisord/log/nginx.err.log).

    Also, you need to check the Nginx log from the APP server (/supervisord/log/nginx.err.log).

    You can link the error in the BP event log with the error in the Nginx error log using the date-time when the error occurred. For example, Nginx sends a 502 bad gateway error to CT (you can see it in the BP event log), but in the Nginx error log, this error looks like 111: Connection refused.

Issue escalation

Before escalating the issue, collect the following information:

  • Current investigation results

  • Control Tower version

  • VDS server version

  • Information about the model: version, type (IE, classification), custom or not

  • BP event log with errors

  • Logs from the VDS server:

    • \<INSTALL_DIR>/supervisord/log/vds-mesos-adapter.out.log
    • \<INSTALL_DIR>/supervisord/log/vds-scaling-service.out.log
    • \<INSTALL_DIR>/supervisord/log/vds-gateway-service.out.log
    • \<INSTALL_DIR>/supervisord/log/nginx.out.log
    • \<INSTALL_DIR>/supervisord/log/nginx.err.log
    • \<INSTALL_DIR>/supervisord/log/marathon.err.log
    • \<INSTALL_DIR>/supervisord/log/mesos-master.err.log
    • Stdout and stderr logs from the Sandbox of a Mesos job

Appendix

← PreviousNext →