Edit

Collect and view logs

This article describes basic concepts of log aggregation.

The following log level options are available:

• prod: logs contain only error messages and warnings.
• dev: logs contain error messages, warnings, info, and debug messages.

The default log level is prod unless the user specifies the dev option before the installation. A user can also change the log level for a particular server or a component, except for RPA and BI servers. Learn more in Change a component's log level.

The solution for logs aggregation consists of the following components:

• Kibana with logs filtering dashboards
• Elasticsearch for storing logs data
• Logstash for aggregation and forwarding log records

The log aggregation process is the following:

1. A service writes logs in the JSON format and places them next to itself on the filesystem.

   Note: the JSON log files are currently supported only for the following components: bep services, bep workers (ct and automl), automl-services, and control-tower.

2. The Filebeat component reads the JSON logfile, parses it, and sends to Logstash.

3. Logstash reads the parsed JSON and sends it to:

   • Elasticsearch in the ES format with particular fields.

   • A separate logfile in a human-readable plain-text format defined by the Logstash configuration.

     By default, IA Cloud Enterprise stores all components' logs on the server with installed ELK (in most cases, the Master server) in the INSTALL_DIR/shared/logs/<component-name>/<hostname>/<date>/ directory as filename.log.

     Element	Description	Example
     install-dir	A directory to place user files, logs, and configs	/opt/workfusion
     component	A component's name	wfagent, workfusion, nginx
     hostname	A real hostname of the server that sends logs	master.workfusion.com
     date	A date when a log event was created on the source server. Pattern: YYYY-MM-DD. The date is created based on UTC and thus may differ from the timezone configured on the server.<	2018-11-30
     filename	A log filename	metrics.log, status-agent-error.log

     Example: /opt/workfusion/shared/logs/nginx/myserver-ct1.workfusion.com/2020-02-13/nginx_error.log.

All logs remain both on the source and the Master server but with different retention policies.

important

You can also see logs of the services that have failed to start and initialize the logging. For more information on browsing logs, see Search a specific message in logs.

Collect logs from all services

You can also collect all logs from services. As a result, the records are archived in the ZIP format, contained in the aggregated_logs_dir directory, and organized as follows:

aggregated_logs_dir > hostname > service > logs

Example:

$ ll /opt/workfusion/shared/logs/tmp-wfan-10024-master1.wflab.io/automl-gateway-service
-rw-rw-r--. 1 wfuser wfuser 8035 Nov  8 11:48 automl-gateway-service-2021-11-04.0.log.json.zip
-rw-rw-r--. 1 wfuser wfuser  638 Nov  5 07:56 automl-gateway-service-startup.log.zip
-rw-rw-r--. 1 wfuser wfuser 6359 Nov  5 07:57 automl-gateway-service.log.json.zip

To collect all service logs from all Linux hosts, on any MASTER server, run the following commands:

$ export ANSIBLE_PRIVATE_KEY_FILE='/path/to/ssh-private-key'
$ export ANSIBLE_VAULT_PASS='your-ansible-vault-pass'

# default installation dir:
$ cd /opt/workfusion/wf_installer
$ ./install.sh configure logs

To collect all service logs from a given host, on any MASTER or AGENT server, run the following commands:

# default tools dir:
$ cd /opt/workfusion/tools
$ ./collect-linux-logs.sh