Edit

Collect and view logs

This article describes basic concepts of log aggregation.

The following log level options are available:

• prod: logs contain only error messages and warnings
• dev: logs contain error messages, warnings, info, and debug messages

The default log level is prod unless the user specifies the dev option before the installation. A user can also change the log level for a particular server or a component, except for RPA and BI servers. Learn more in Change a component's log level.

The solution for logs aggregation consists of the following components:

• Kibana with logs filtering dashboards
• Elasticsearch for storing logs data
• Logstash for aggregation and forwarding log records

The log aggregation process is the following:

1. A service writes logs in the JSON format and places them next to itself on the filesystem.

   Note: the JSON log files are currently supported only for the following components: bep services, bep workers (ct and automl), automl-services, and control-tower.

2. The Filebeat component reads the JSON logfile, parses it, and sends to Logstash.

3. Logstash reads the parsed JSON and sends it to Elasticsearch in the ES format with particular fields.

   Element	Description	Example
   component	A component's name	wfagent, workfusion, nginx
   hostname	A real hostname of the server that sends logs	master.workfusion.com
   date	A date when a log event was created on the source server. Pattern: YYYY-MM-DD. The date is created based on UTC and thus may differ from the timezone configured on the server.<	2018-11-30
   filename	A log filename	metrics.log, status-agent-error.log

   Example: nginx/myserver-ct1.workfusion.com/2020-02-13/nginx_error.log.

All logs remain both on the source and the Master server but with different retention policies.

important

You can also see logs of the services that have failed to start and initialize the logging. For more information on browsing logs, see Search a specific message in logs.