Edit

Collect and view logs

This article describes basic concepts of log aggregation.

The following log level options are available:

• prod: logs contain only error messages and warnings
• dev: logs contain error messages, warnings, info, and debug messages

The default log level is prod unless the user specifies the dev option before the installation. A user can also change the log level for a particular server or a component, except for RPA and BI servers. Learn more in Change a component's log level.

The solution for logs aggregation consists of the following components:

• Kibana with logs filtering dashboards
• Elasticsearch for storing logs data
• Logstash for aggregation and forwarding log records

The log aggregation process is the following:

1. A service writes logs in the JSON format and places them next to itself on the filesystem.

   Note: the JSON log files are currently supported only for the following components: bep services, bep workers (ct and automl), automl-services, and control-tower.

2. The Filebeat component reads the JSON logfile, parses it, and sends to Logstash.

3. Logstash reads the parsed JSON and sends it to Elasticsearch in the ES format with particular fields.

   Element	Description	Example
   component	A component's name	wfagent, workfusion, nginx
   hostname	A real hostname of the server that sends logs	master.workfusion.com
   date	A date when a log event was created on the source server. Pattern: YYYY-MM-DD. The date is created based on UTC and thus may differ from the timezone configured on the server.<	2018-11-30
   filename	A log filename	metrics.log, status-agent-error.log

   Example: nginx/myserver-ct1.workfusion.com/2020-02-13/nginx_error.log.

All logs remain both on the source and the Master server but with different retention policies.

important

You can also see logs of the services that have failed to start and initialize the logging. For more information on browsing logs, see Search a specific message in logs.

Collect logs from all services

You can also collect all logs from services. This can be useful in case of incidents when you need to send the logs bundle to the WorkFusion Support. As a result, the records are archived in the ZIP format, contained in the common_nfs_share/logs directory, and organized as follows:

common_nfs_share > logs > hostname > service > logs

Example:

$ ll /opt/workfusion/shared/logs/tmp-wfan-10024-master1.wflab.io/automl-gateway-service
-rw-rw-r--. 1 wfuser wfuser 8035 Nov  8 11:48 automl-gateway-service-2021-11-04.0.log.json.zip
-rw-rw-r--. 1 wfuser wfuser  638 Nov  5 07:56 automl-gateway-service-startup.log.zip
-rw-rw-r--. 1 wfuser wfuser 6359 Nov  5 07:57 automl-gateway-service.log.json.zip

To collect all service logs from all Linux hosts, on any MASTER server, run the following commands:

$ export ANSIBLE_PRIVATE_KEY_FILE='/path/to/ssh-private-key'
$ export ANSIBLE_VAULT_PASS='your-ansible-vault-pass'

# default installation dir:
$ cd /opt/workfusion/wf_installer
$ ./install.sh configure logs

To collect all service logs from a given host, on any MASTER or AGENT server, run the following commands:

# default tools dir:
$ cd /opt/workfusion/tools
$ ./collect-linux-logs.sh