Collect and view logs
This article describes basic concepts of log aggregation.
Overview
The following log level options are available:
- prod — logs contain only error messages and warnings
- dev — logs contain error messages, warnings, info and debug messages
The default log level is prod, unless user specifies the dev option before the installation. A user can also change the log level for a particular server or a component, except for RPA and BI servers. Learn more in Change a component's log level.
The solution for logs aggregation consists of the following components:
- Kibana with logs filtering dashboards
- Elasticsearch for storing logs data
- Logstash for aggregation and forwarding log records
The log aggregation process is the following:
A service writes logs in the .json format and places them next to itself on the filesystem.
Note: the JSON logfiles are currently supported only for the following components: bep sevices, bep workers (ct and automl), automl-services, and control-tower.
The Filebeat component reads the .json logfile, parses it, and sends to Logstash.
Logstash reads the parsed JSON and sends it to:
Elasticsearch in the ES format with particular fields.
A separate logfile in a human readable plain-text format defined by the Logstash configuration.
By default, IA Cloud Enterprise stores all components' logs on the server with installed ELK (in most cases, the APP server) in the INSTALL_DIR/shared/logs/<component-name>/<hostname>/<date>/ directory as filename.log.
Element Description Example install-dir A directory to place user files, logs, and configs /opt/workfusion component A component's name wfagent, workfusion, nginx hostname A real hostname of the server that sends logs int.workfusion.com date A date when a log event was created on the source server. Pattern: YYYY-MM-DD. The date is created based on UTC time and thus may differ from the timezone configured on the server.< 2018-11-30 filename A log file name metrics.log, status-agent-error.log Example: /opt/workfusion/shared/logs/nginx/myserver-ct1.workfusion.com/2020-02-13/nginx_error.log.
All logs remain both on source and the APP server but with different retention policies.
important
You can also see logs of the services that have failed to start and initialize the logging. For more information of browsing logs, see Search a specific message in logs.