Skip to main content

Issues related to application performance

The article contains instructions on addressing manual step issues caused by improper Control Tower (CT) or Workspace (WS) performance. Read respective sections below to verify the symptoms, identify the root cause, and find a solution.

Control Tower performance​

Possible symptoms of CT performance issues are as follows:

  • Slow submission of hits from CT

  • Slow execution of CT jobs, such as streaming, renewable hit processing, hit sorting, and so on

To investigate, follow the steps below:

Check performance metrics​

  • Check if the issue is reproduced at the time of the investigation. Go to Kibana and review the Cluster Utilization dashboard for any traces of the issue.

  • Go to the System Overview tab to examine the overall system performance host-by-host and identify which components have the heaviest workload.

  • In Kibana, check the Host Overview dashboard to evaluate the CPU utilization, memory consumption, and network traffic intensity.

tip

You may also want to read Issues related to CT performance, but when you don't find answers, escalate the issue to the Support team.

Review logs​

In Kibana, check the following logs for slow application performance:

  • Performance metrics log for Control Tower
  • Nginx log
  • Workspace logs
  • Control Tower logs

Use Discover, expand logs, and set filters to quickly find what you are looking for. Go to specific application logs, pick up the timeframe. If needed, set an additional filter for the host you want to analyze:

Then, carefully review the metricbeat and cluster.metrics logs:

The logs can look similar to the example below and provide lots of useful information, such as:

  • Process description

  • Event time and duration

  • Involved resources and so on

@timestamp                          Oct 18, 2021 @ 12:29:34.453
@version                            1
_id                                 1X7zknwBhBHF1ndnEVZd
_index                              metricbeat-7.5.2-2021.10.18
_score                              - 
_type                               _doc
agent.ephemeral_id                  af73ea40-425d-47de-9a60-2603d20a1e26
agent.hostname                      ip-10-100-1-84.ec2.internal
agent.id                            a5a03034-d0f3-41c4-9787-afcb7fa6ebae
agent.name                          wfinst-978-wfaw-10033-ct1.workfusion.com
agent.type                          metricbeat
agent.version                       7.5.2
ecs.version                         1.1.0
event.dataset                       system.process
event.duration                      146,215,593
event.module                        system
fields.ns                           int
host.name                           wfinst-978-wfaw-10033-ct1.workfusion.com
metricset.name                      process
metricset.period                    30000
process.pgid                        3,839
process.pid                         4,358
process.ppid                        3,839
process.working_directory           /opt/workfusion/zookeeper/datalog/version-2
service.type                        system
system.process.cpu.start_time       Oct 18, 2021 @ 11:21:52.000
system.process.cpu.total.norm.pct   0.033
system.process.cpu.total.pct        0.261
system.process.cpu.total.value      425,560
system.process.fd.limit.hard        200,000
system.process.fd.limit.soft        200,000
system.process.fd.open              29
system.process.memory.rss.bytes     16,347,136
system.process.memory.rss.pct       0.001
system.process.memory.share         1,843,200
system.process.memory.size          461,180,928
system.process.state                running
tags                                beats_input_raw_event
user.name                           wfuser

Workspace performance​

Possible symptoms of issues in WS performance include slow assignment acceptance and submission and task handling.

To investigate, check the localhost-access logs for Workspace. Determine whether there are delays in user actions on the WS side and compare them to Nginx logs.

To correlate the slowdowns with user activity, read incoming request logs and match incoming tasks with events and responses.

Example 1: a user logged in and was redirected to the dashboard controller.

127.0.0.1 - - [30/Jul/2021:11:56:49 +0000] "GET /workspace/login?ldapConfigured=false&samlEnabled=true HTTP/1.1" 200 4648
127.0.0.1 - - [30/Jul/2021:11:56:50 +0000] "POST /workspace/saml/SSO HTTP/1.1" 302 -
127.0.0.1 - - [30/Jul/2021:11:56:50 +0000] "GET /workspace/login?ldapConfigured=false&samlEnabled=true HTTP/1.1" 302 -
127.0.0.1 - - [30/Jul/2021:11:56:50 +0000] "GET /workspace/dashboard HTTP/1.1" 200 36605

Example 2: a user accesses a specific task and opens it with the auto-acceptance checkbox.

127.0.0.1 - - [30/Jul/2021:12:10:24 +0000] "GET /workspace/task/CVBQ9ZDF014RKBNXYP8511ADNCHIVKBBO?autoAccept=true&ldapConfigured=false&samlEnabled=true HTTP/1.1" 200 32775
127.0.0.1 - - [30/Jul/2021:12:10:24 +0000] "GET /workspace/categories HTTP/1.1" 200 25245

The task submission log looks like this:

127.0.0.1 - - [30/Jul/2021:12:10:24 +0000] "POST /workspace/submit HTTP/1.1" 200 100

From the second end, in Nginx logs, you can track when requests are sent to Workspace API, as well as delays, bad request statuses, security, or other issues.

tip

Align component timezones to have a good tracking position and absolute transparency in the environment. Check component configurations and make sure there is no endpoint cross-mapping, especially when working with multiple WS and CT instances.

Analyze output​

After confirming any of the issues above, gather the following statistics and escalate them to the Support team:

  • Kibana dashboard diagram for incoming or processed tasks over the incident time interval

  • Kibana System overview dashboard over the incident time interval

  • Error message traces from Kibana

  • Control Tower and Workspace uptime

If you fail to confirm the issues, continue the investigation.

View also: