Change log level for components

Message logging in applications is based on logger objects. Each logger has the following attributes:

• The name attribute is mandatory and is used to specify the logger context. Usually, it is defined as a package name or full name of the class that writes logs.
• The level attribute is optional and is used to specify the logger scope.

Log levels

Below is a list of log levels arranged in the order of increasing their scope:

• OFF
• ERROR
• WARN
• INFO
• DEBUG
• TRACE

A request to log a message is enabled if the log level is the same as or higher than the level of the logged message.

The following table shows if the request to log a message with a particular level (row) is enabled or disabled by the logger with a particular level (column).

Logged message level	Effective log level
	OFF	ERROR	WARN	INFO	DEBUG	TRACE
ERROR	NO	YES	YES	YES	YES	YES
WARN	NO	NO	YES	YES	YES	YES
INFO	NO	NO	NO	YES	YES	YES
DEBUG	NO	NO	NO	NO	YES	YES
TRACE	NO	NO	NO	NO	NO	YES

Logging hierarchy

Loggers have a hierarchical structure that closely resembles that of a Java object. All loggers are descendants of a predefined root logger.

There are two rules describing the logging hierarchy:

• A logger is an ancestor when its name, followed by a dot, precedes a descendant logger's name.
• A logger is a parent when there are no ancestors between it and its child.

The following example describes the logger-level inheritance:

<root level="DEBUG"/>
<logger name="com.workfusion" level="ERROR">
<logger name="com.workfusion.spa" level="TRACE">

Example of loggers extending the level from configured loggers: root / com.workfusion / com.workfusion.spa:

com.workfusion.Example      - effective level ERROR # com.workfusion.Example is an ancestor of the com.workfusion logger and, therefore, has the same level—ERROR—as the com.workfusion logger
com.workfusion.spa.Example  - effective level TRACE # com.workfusion.spa.Example is an ancestor of the com.workfusion.spa logger and, therefore, has the same level—TRACE—as com.workfusion.spa logger
org.hibernate               - effective level DEBUG # org.hibernate is an ancestor of the ROOT logger and, therefore, has the same level—DEBUG—as root logger

Configuring log level

There are two approaches to configuring the log level:

• At runtime

• With restart

At runtime

This is the recommended approach to change the log level.

For all Workers, except OCR Workers, and some other components runtime logging is enabled by default. For other components, you need to enable runtime logging if you want to use the approach for changing the logging configuration.

The table below lists all components and their default runtime logging settings.

Component name	Application name	Runtime logger enabled by default	Runtime logger can be applied
Control Tower	control-tower	true	true
WorkSpace	workspace	true	true
Manual Task Renderer	manual-task-renderer	true	true
Worker Management Service	worker-management-service	false	true
Task Dispatcher Service	task-dispatcher-service	false	true
Hazelcast server	hazelcast-server	false	false
Bot Manager	bot-manager	false	true
Bot Agent	bot-agent	false	true
OCR REST	ocr-rest	false	true
AutoML Billing Service	automl-billing-service	false	false
AutoML Gateway Service	automl-gateway-service	false	true
AutoML Model Service	automl-model-service	false	true
AutoML Model Management Service	automl-model-management-service	false	true

Enable runtime logging

To enable runtime logging for a specific component, follow the steps below:

1. Check the table above to see whether the runtime logging is enabled for the component by default (see the Runtime logger enabled by default column).

   • If enabled, you can skip the rest of the steps and proceed to configure the log level at runtime.

   • If not enabled, see Step 2.

2. In ZooKeeper, check if the /config/{app_name}/management.logging.enabled {true|false} property is available and what value it has.

   • If the property is set to false, set it to true.

   • If there's no such property, create it using the command:

     create /config/{app_name}/management.logging.enabled {true|false}

3. Restart the component.

Change log level at runtime

There are two options how you can change the log level at runtime:

• For a specific package or entire application

• Profile approach

For specific package or entire application

In this case, zNode in ZooKeeper is interpreted as a logger name and its value—as the logger's level. The application reacts to all changes of zNodes and changes the log level accordingly.

To configure the log level for a specific package, set the /config/{app_name}/log/{package_name} ZooKeeper zNode to a log level value (case insensitive), for example, TRACE, DEBUG, INFO, WARN, ERROR. In this zNode:

• {app_name} is the name of the Spring Boot application. Typically, you can find it in Marathon.

• {package_name} is the package name, for example, com.workfusion.

To configure the log level for the entire application, set {package_name} to root.

To set the ZooKeeper zNode and change the log level, log in via SSH to the Master server and run the following commands as WFUSER:

$ cd /opt/workfusion/zookeeper/bin
$ ./zkCli.sh -server {server_name}  # where {server_name} is the ZooKeeper host and port. The default value is 127.0.0.1:2181.

# Get a list of nodes for the component
$ ls /config/{app_name}     # where {app_name} is the service name of the component.

# Verify that the /config/{app_name}/log directory is present. If it is absent while the application is running, most likely, the runtime configuration is disabled for the app.

# Create a node with a desired package and log level
$ create /config/{app_name}/log/{package_name} {log_level}

# In case the /config/{app_name}/log/{package_name} node exists, check its value and update if needed.

$ get /config/{app_name}/log/{package_name}
$ set /config/{app_name}/log/{package_name} {log_level}

# Examples:
$ ls /config/worker-ct # Output contains the 'log' child node.
$ create /config/worker-ct/log/com.workfusion.worker.ct.bcb DEBUG   # This changes the log level to DEBUG for all classes under the 'com.workfusion.worker.ct.bcb' package. Changes are automatically applied to all ct-workers with no restart required.

Deleting the /config/{app_name}/log/{logger_name} zNode leads to implicit resetting of the log level to its parent log level.

info

A restart is not needed. The changes are applied at runtime.

Profile approach

Under this approach, log levels are grouped as logging profiles (for example, prod or dev) and stored within application sources as a properties file.

note

The approach is applicable only for applications implemented with multiple profiles. For example, for Control Tower, profiles can be prod and dev.

The application reacts to the updates of a configured zNode in ZooKeeper. When the node value changes, the application triggers the log level reconfiguration according to the received logging-profile-id.

For some components (for example, Control Tower), when you change the log level, restart is needed. For most components, the changes are applied at runtime.

Log levels in logback.xml (logback-spring.xml) must be set as placeholders. The files contain the logger name and its level. To configure additional loggers, add them to logback.xml (logback-spring.xml).

To change a component's log level, log in via SSH to the Master server and run the following commands as WFUSER:

$ cd /opt/workfusion/zookeeper/bin
$ ./zkCli.sh -server {server_name}  # where {server_name} is the ZooKeeper host and port. The default value is 127.0.0.1:2181.

# Get a list of nodes in the component's log directory
$ ls /config/{app_name}/log     # where {app_name} is the service name of the component.

# If the /config/{app_name}/log directory contains no node named logprofile, create it with an appropriate profile identifier.

$ create /config/{app_name}/log/logprofile {logging-profile-id}

# Supported values for {logging-profile-id}: dev, prod

# In case the /config/{app_name}/log directory includes a node named logprofile, check its value and update if needed.

$ get /config/{app_name}/log/logprofile
$ set /config/{app_name}/log/logprofile {logging-profile-id}

# Examples:
$ create /config/control-tower/log/logprofile dev   # Create a logprofile node and set its value to dev, which updates the log level of appropriate loggers in Control Tower.
$ set /config/control-tower/log/logprofile prod     # Update the logprofile node and set its value to prod, which updates the log level of the loggers in Control Tower.

tip

In the example above, values in {} are placeholders. Change them appropriately.

info

A restart is not needed. The changes are applied at runtime.

With restart

This is the standard configuration approach for all Java applications, whereby you can change the log level using one of the three options:

• Option 1: for those applications that are self-contained artifacts, such as Spring Boot-based applications, you can configure the log level via ZooKeeper's cloud-config-zookeeper. In this case, applications will take up the log level configurations from ZooKeeper at startup.

  To change the log level using the above method, log in via SSH to the Master server and run the following commands as WFUSER:

  $ cd /opt/workfusion/zookeeper/bin
  $ ./zkCli.sh -server {server_name}  # where {server_name} is the ZooKeeper host and port. The default value is 127.0.0.1:2181.
  
  # Get a list of nodes in the component's properties directory
  # ls /config/{app_name}         # where {app_name} is the service name of the component.
  
  # If the /config/{app_name} directory contains no node with the same name as the package with the logging.level. prefix (for example, logging.level.{logger_name}), create it with an appropriate log level.
  
  $ create /config/{app_name}/logging.level.{logger_name} {log_level}
  
  # Supported values for {logger_name} are ROOT and any other logger name.
  # Supported values for {log_level} are listed in the order of increasing the logger scope: OFF, ERROR, WARN, INFO, DEBUG, TRACE.
  
  # If the /config/{app_name} directory includes a node named as logging.level.root or logging.level.{logger_name}, check its value and update if needed.
  
  $ get /config/{app_name}/logging.level.{logger_name}
  $ set /config/{app_name}/logging.level.{logger_name} {log_level}
  
  # Examples:
  $ create /config/ocr-rest/logging.level.root INFO       # Create the logging.level.root node and set its value to INFO.
  $ set /config/ocr-rest/logging.level.root WARN          # Update the logging.level.root node and set its value to WARN.
  tip

  In the example above, the values in {} are placeholders. Change them appropriately.

• Option 2: change the application's startup command line. For example, add -Dlogging.level.{logger_name}={log_level} as a JVM parameter in the application's startup command line.

• Option 3: Change the application.properties files for those applications where it’s possible (for CT, change the workfusion.properties files). For example, add logging.level.{logger_name}={log_level} in the configuration file.

caution

Whichever of the above options you choose to change the log level, restart the component for the changes to be applied.

Default log level values

Click to see default log levels

Control Tower

There are two profiles in Control Tower—prod and dev. The default profile is prod.

Logger name	prod profile level	dev profile level
root	error	error
com.workfusion.utils.logger.ConfigLogger	warn	debug
com.workfusion.spa.core.execution.client.QueueTaskChannel	debug	debug
com.freedomoss.spring.HsPreCleanLocalContainerEntityManagerFactoryBean	warn	debug
com.workfusion.secretmanagement.integration.SecurePropertyFieldCallback	warn	debug
com.workfusion.controltower.spring.ExposableResourceBundleMessageSource	warn	debug
org.springframework.security.saml.metadata.MetadataManager	warn	debug
org.springframework.security.saml.util.SLF4JLogChute	warn	debug
com.workfusion.util.healthcheck	warn	debug
com.workfusion.scheduled.quartz.SchedulerFactoryWithCleanDBBean	warn	debug
com.workfusion.api.service.healthcheck.impl	warn	debug
com.workfusion.api.controller.HealthCheckRestController	warn	debug
com.freedomoss	warn	debug
com.workfusion	warn	debug
audit	warn	debug
com.freedomoss.crowdcontrol.webharvest	warn	debug
com.freedomoss.crowdcontrol.webharvest.plugin.recorder	warn	debug
com.workfusion.studio.rpa.recorder.model	warn	debug
org.webharvest.runtime.ConfigExecutionLogger	warn	trace
com.freedomoss.crowdcontrol.webharvest.plugin.selenium	warn	debug
com.freedomoss.crowdcontrol.webharvest.policy.ThrottlingExecutionPolicy	warn	debug
com.freedomoss.objective.impl.CompositeCampaignResultProcessorImpl	warn	info
com.freedomoss.objective.impl.BatchedCompositeStepTransitionService	warn	info
com.freedomoss.objective.impl.RuleUtils	warn	debug
com.freedomoss.requester.service.run.impl.StreamService	warn	info
com.freedomoss.crowdcontrol.listener.UserSessionListener	warn	debug
com.freedomoss.workfusion.web.security.UserLoginSuccessHandler	warn	debug
org.apache.cxf.services	warn	info
com.freedomoss.objective.common.SharedData	info	info
com.freedomoss.requester.service.mes.impl.EvenlyStrategy	warn	trace
com.freedomoss.requester.service.mes.impl.AdvancedStrategy	warn	trace
com.freedomoss.crowdcontrol.service.ScheduledCampaignUploadDataService	warn	debug
com.freedomoss.crowdcontrol.webharvest.ScheduledCampaignExecutionBean	warn	debug
com.freedomoss.crowdcontrol.webharvest.ScheduledCampaignExecutionBean$ScheduleRunTask	warn	debug
com.freedomoss.workfusion.performance.PerformanceMonitor	warn	debug
org.hibernate.SQL	warn	warn
com.freedomoss.jobber	warn	debug
com.freedomoss.jobber.web_harvest	warn	debug
com.workfusion.concurrency.stripes	warn	debug
com.workfusion.web.controllers	warn	debug
com.workfusion.ws	warn	debug
org.apache.cxf.interceptor	warn	info
org.apache.cxf.interceptor_2	error	error
org.springframework.security.saml	warn	debug
com.freedomoss.requester.service.project.impl.matching	warn	debug
com.freedomoss.objective.impl.RuleExecutor	warn	info
org.quartz.plugins.history.LoggingTriggerHistoryPlugin	error	error
org.quartz.plugins.history.LoggingJobHistoryPlugin	error	error
com.freedomoss.objective.impl.WorkspaceClientAdapter	warn	debug
com.freedomoss.objective.impl.HitApplicationService	warn	debug
com.workfusion.controltower.core.api.logging.ApiLoggingFilter	warn	debug
com.freedomoss.requester.service.automation	warn	debug
com.freedomoss.requester.service.automation.util.AutomationUtil	warn	debug
com.freedomoss.requester.util.CampaignCopyUtil	warn	debug
com.freedomoss.crowdcontrol.controller.CampaignController	warn	debug
com.freedomoss.crowdcontrol.controller.job.AutomationRecommendationController	warn	debug
com.freedomoss.crowdcontrol.controller.job.JobController	warn	debug
com.freedomoss.requester.service.task.impl.TaskRecommendationService	warn	debug
com.freedomoss.requester.service.xml.impl.XmlImporterService	warn	debug
com.freedomoss.requester.service.task.impl.HitService	warn	debug

WorkSpace

There are two profiles in WorkSpace—prod and dev. The default profile is prod.

Logger name	prod profile level	dev profile level
root	error	error
com.freedomoss.virtualizer	warn	info
org.springframework.jdbc.core	warn	warn
org.springframework.web.client	warn	info
org.springframework.security	warn	warn
org.apache.cxf.interceptor.LoggingInInterceptor	warn	warn
org.apache.cxf.interceptor.LoggingOutInterceptor	warn	warn
org.apache.cxf	warn	warn