Search specific message in logs

The article describes the necessary steps to search and filter particular messages in selected Kibana logs.

You can search through logs via three different ways:

• Dashboard

  The simplest and most convenient way to find the latest logs of a particular service. Dashboards already contain commonly used fast filters to narrow the search.

• Discover

  A more sophisticated and customizable way. Here, you can refine your search and find BEP logs and records from other services. You can create filters on the fly. The search field supports Kibana Query Language (KQL).

• Logs UI

  Useful for reading logs in the realtime. Logs UI provides a stream of logs and the alternative temporal navigation for logs. The search field supports Kibana Query Language (KQL).

tip

For additional details not mentioned in this document refer to the official Kibana guides:

• Using search
• Discover
• Logs UI

Dashboard

To search a message via the WorkFusion dashboard in Kibana, follow the steps below:

1. Open the Dashboard page.

2. Open [WorkFusion IA Cloud] Application Logs dashboard.

   

3. In the Search field, enter a message to search, for example, "automl-gateway-service" and then click Refresh.

   For more details on search queries, see Kibana queries and filters.

   

   After you find the required log, expand it to view the details.

4. To view the context of the log, click View surrounding documents.

   

5. To view the whole logfile, not just a message in it, clear the message box, specify Hostname, Service name, and Log name shown in the details of the log, and then click Apply changes.

   

6. To change the filtering time range, open the Time-range menu, and choose the time interval for logs filter.

   

Discover

To search a log message via Discover, follow the steps below:

1. In the main menu, click Discover.

   

2. Select the filebeat-* index of all running services. You can use the index if the above dashboard doesn't have the necessary functionality.

3. Configure the fields to be displayed in the log table. For that, in the Available fields group, point to a required field, and click the table icon to select it.

   

   We recommend to add the following fields:

   • level
   • message
   • service_name in case you want to browse logs for several services

   

4. If needed, apply the required filter to narrow the results.

   When searching BEP logs, you can filter the worker logs by the app-name field, for example, Control Tower or AutoML.

Logs UI

To search a log message in Logs UI, follow the steps below:

1. In the main menu, click Logs UI. The log stream appears.

   

2. In the Search field, specify your query. Here, you can use the Kibana Query Language for search and result refinement.

   You can use the following parameters in most cases:

   Field	Description	Example
   service_name.keyword	Name of the service	workspace, automl-gateway-service
   beat.hostname.keyword	Hostname or FQDN of server	example.domain.com
   level	Logging level. Possible values: ERROR WARN INFO FATAL DEBUG TRACE For more information on the log levels, see Change log level for components.	level: ERROR

note

For browsing messages with stake trace (filter stake_trace: *), it is recommended to use the Discover way for more structured formatting.

Filter logs

To filter logs of a particular service, follow the steps below:

1. In the Filter panel, click Add a filter. The Add filter window appears.

2. In the Filter group, in the Fields box, select service_name.keyword. In the Operators box, select is; and in the Values box, select the required service.

   

3. Click Save.

You can apply the following variables as filters:

List of services

The list of services can be used for filtering by the service_name field.

CT apps

• workfusion
• workspace
• hazelcast-server
• sqc

BEP

• task-dispatcher-service
• worker-management-service
• bep-worker

AutoML

• automl-gateway-service
• automl-model-management-service
• automl-model-service

RPA

• bot-manager
• bot-agent

OCR

• ocr-rest
• ocr-worker
• ocr-task11
• ocr-task12

Third-party components and services

• marathon
• mesos-master
• mesos-agent
• rabbitmq
• s3
• nexus
• vault
• supervisord
• zookeeper
• zoonavigator
• mssql

Running and failover

• supervisord
• failover

Proxy

• nginx
• haproxy
• haproxy-tcp

ELK

• elasticsearch
• heartbeat
• kibana
• logstash
• metricbeat

Logging variables

Components with structured JSON-based logs

• level: log level (DEBUG, INFO, WARN, ERROR, and so on)
• logger_name: logger name
• thread_name: thread name
• stack_trace: stack-trace (if present)
• caller_class_name: name of the class where a log event emitted (in most cases, it is the same as logger_name)
• caller_file_name: filename of the class
• caller_line_number: line number on which a log event emitted
• caller_method_name: method in which a log event emitted

Components with plain-text logs

• level: log level, parsed (where possible) from a message

control-tower

• ct_bp_name: Business Process name
• ct_bp_uuid: Business Process UUID
• ct_step_id: step ID
• ct_step_name: step name
• ct_task_id: task ID (or hit ID)
• ct_odf_transaction: ODF transaction ID

task-dispatcher-service

• bep_task_id: BEP task UUID
• bep_task_queue_name: task queue name
• bep_worker_queue_name: worker queue name
• Any task meta-data passed with the addMetadataWithLogging() method

worker-management-service

• bep_client_app_id: BEP client app ID
• bep_worker_gavp: worker GAVP
• bep_worker_group: worker group

bep-worker

• Any task meta-data passed with the addMetadataWithLogging() method

automl-gateway-service

• automl_model_id: trained model ID
• automl_artifact_id: artifact ID (model-code:version)
• automl_training_id: training ID

automl-model-service

• automl_model_id: trained model ID
• automl_artifact_id: artifact ID (model-code:version)
• automl_training_id: training ID

automl-model-management-service

• automl_model_id: trained model ID
• automl_artifact_id: artifact ID (model-code:version)

nginx

• http_method: HTTP methods (GET, POST, HEAD, and so on).
• http_request: request path.
• http_request_total_time: total request time: from the request received to the last byte sent to the client.
• http_status: HTTP status code (200, 404, 503, and so on).
• http_upstream_header_time: upstream service request processing time: from the request sent to the upstream first byte of the response received from upstream. Usually, it is the request process time by the upstream service. For details on Nginx times, see the documentation.
• http_upstream_response_time: upstream service response complete time: from the request sent to the upstream last byte of the response received from upstream. For details on Nginx times, see the documentation.
• http_user_agent: user agent (browser or library identification).

haproxy

• haproxy_upstream_service: the name of upstream service (request-target service)
• http_status: HTTP status (only of HTTP upstreams)

\<service>-access

For tomcat-running apps, such as workfusion-access, workspace-access, and sqc-access:

• http_method: HTTP methods (GET, POST, HEAD, and so on)
• http_request: request path
• http_status: HTTP status code (200, 404, 503, and so on)

note

For an event with http_status, the following levels are set:

• DEBUG: for 1xx, 2xx statuses
• WARN: for 4xx statuses
• ERROR: for 5xx statuses

Examples

Most of the examples below can be combined.

What to find	UI filter	Kibana Query Language filter
Search by custom MDC variable	mdc_bep_task_id is edf573c9-7bcd-4d11-91c7-e21a4c140d57	mdc_bep_task_id: "edf573c9-7bcd-4d11-91c7-e21a4c140d57"
Only logs event with stack-trace	stack_trace exists	stack_trace: *
Only error log events	level is ERROR	level: ERROR
Logs from specific node	agent.name is HOST	agent.name: "HOST"
Logs for specific class	caller_class_name is com.workfusion.spa.core.service.task.dispatcher.cleanup.Utils	caller_class_name: com.workfusion.spa.core.service.task.dispatcher.cleanup.Utils
Log for specific services	service_name is one of task-dispatcher-service, worker-management-service	service_name: ("task-dispatcher-service" or "worker-management-service")
Log for specific service	service_name is task-dispatcher-service	service_name: "task-dispatcher-service"
Error or warn log event	level is one of ERROR, WARN	level: (ERROR or WARN)

info

When using the Kibana Query Language in filters, remember to use quotation marks ("") with the query to explicitly find the exact match.

Export search results

You can export the search results from the Discover page as the JSON file.

To export the data, follow the steps below:

1. Find the required logs via Discover by adding filters, setting the required period, and so on.

2. Click the Inspect tab. The Inspect sidebar appears.

3. On the Inspect sidebar, click Response.

4. Click the Copy icon in the upper-right corner. The response in the JSON format is copied to the clipboard.

   

5. Paste the response to a text file and save it as a JSON file.