Configure log rotation

This article explains advanced configuration of installer to control log rotation and retention.

Logs rotation policies

The following policies control the rotation and expiration of logs on the PM server in INSTALL_DIR/logs directory:

• Size-based log rotation for current date.

  When an original log file reaches its maximum, it is rotated and the suffix is added.

  • Number of rotated files: 7. Parameter logs_rotate_count in PACKAGE_DIR/group_vars/all/vars.yml).

  • Maximum size of a file: 200MB. Parameters logs_rotate_maxsize and logs_rotate_size in PACKAGE_DIR/group_vars/all/vars.yml).

    If the log file size exceeds the Maximum size * Number of rotated files (by default 1.4GB), log data is removed from the central server, but remains on the source server.

• Log compression.

  All files older than today will be automatically compressed with gunzip.

• Log cleanup.

  All files older than 14 days will be automatically removed from the aggregation server. This behaviour is configured in installer's PACKAGE_DIR/group_vars/all/vars.yml file, in parameter logs_retention, before installation of the Master server. If not specified, the default value is 13, which means that files older than 14 days are removed.

Log rotation configuration

These parameters are optional during the configuration process, but they may be used to control log retention in case of specific data storage policies.

• logs_retention

  The number of rotated files of the same log file. Default: 7.

  For manual reconfiguration, edit INSTALL_DIR/logrotate/logrotate.d/<service_name>.conf:

  $ vi /opt/workfusion/logrotate/logrotate.d/<service_name>.conf # under wfuser
  
  # Modify line below
  ...
  rotate 7
  ...

• logs_rotate_maxsize

  The size of each rotated file. Default: 200M.

  For manual reconfiguration, edit INSTALL_DIR/logrotate/logrotate.d/<service_name>.conf:

  $ vi /opt/workfusion/logrotate/logrotate.d/wf_aggregated_logs.conf # under wfuser
  
  # Modify both lines below
  ...
  size 200M
  ...

Service logs configuration

Rotation is performed daily, or when an original log file reaches its maximum. Logs are deleted, if total capacity of the archived files exceeds 10GB, or they are older than 14 days.

Supported components:

Component Name	Service Name	Configure log level approach
Control Tower	control-tower	Profile
Workspace	workspace	Profile
Workspace API	workspace-api	Profile
Worker Management Service	worker-management-service	Explicit
Task Dispatcher Service	task-dispatcher-service	Explicit
Bot Manager	bot-manager	Explicit
Bot Agent	bot-agent	Explicit
OCR rest	ocr-rest	Explicit
OCR worker	ocr-worker	Explicit
AutoML Gateway Service	automl-gateway-service	Explicit
AutoML Model Service	automl-model-service	Explicit
AutoML Model Management Service	automl-model-management-service	Explicit
AutoQC (sqc)	sqc	Explicit

The following parameters are optional and use the default values, unless specified otherwise.

Parameter	Default value	Description
LOGGING_PROFILE	dev	The logging profile that is used at startup. If ZooKeeper has other profile set, it will be used. Allowed valued: dev, prod.
CONSOLE_LOG_PATTERN, FILE_LOG_PATTERN	Logs pattern: %d{${LOG_DATEFORMAT_PATTERN:-yyyy-MM-dd HH:mm:ss.SSS}} ${LOG_LEVEL_PATTERN:-%-5p} ${PID:- } [%t] %logger{36} : %m%n}	Formatting of logs. For more information about logs pattern see: https://logback.qos.ch/manual/layouts.html#ClassicPatternLayout
LOGS_FILE_PATH	logs	Log files path relative to the installation folder. The full path by default is <install-dir>/&lt;service-name>/logs
LOGS_MAX_FILE_SIZE	100MB	The log file's maximum size. After exceeding the size, an application opens a new log file and writes logs to it. The old file is renamed with the index increased. This value option can be specified in bytes, kilobytes, megabytes, or gigabytes by suffixing a numeric value with KB, MB and respectively GB. Note that extremely small and large values affect the performance.
LOGS_MAX_HISTORY	14	The amount of days during which the logs are stored. The older logs will be deleted.
LOGS_TOTAL_SIZE_CAP	10GB	Max capacity of all log files. If it is exceeds, the oldest logs will be deleted.

Manual reconfiguration

This approach changes values for all services that are installed on a machine.

$ vi /opt/workfusion/environment.sh         # add override of any variable listed below and save changes
...
export LOGS_FILE_PATH=new_logs/
...
$ wfmanager restart <service-name>            # restart service to use new value

To change a value only for one service:

For Control Tower, Workspace, AutoQC (sqc):

$ vi /opt/workfusion/<service-name>/bin/setenv.sh      # open tomcat setenv.sh file and add env variable
...
 -DLOGS_MAX_FILE_SIZE=10MB \
...
$ wfmanager restart <service-name>                                    # restart service to use new value

For other services:

$ vi /opt/workfusion/supervisord/apps/<service-name>.ini   # open supervisord file for service and change
# add to JAVA_OPTS necessary variable
...
JAVA_OPTS="-DLOGS_FILE_PATH=new_logs/"
...
$ wfmanager restart <service-name>                                    # restart service to use new value