Hardware resource exhaustion
Sometimes, slow performance can be due to an exhausted hardware resource or simultaneous exhaustion of multiple resources. Most probably, these resources are CPU, RAM, and IO (disk usage).
To get a better picture of the problem, launch the resource-consuming activity (if you know what it is) and monitor resources at that time.
Top command
Use the command below to get data about the top consumers of CPU or RAM.
[root@app1 ~]# top
top - 11:39:42 up 1 day, 21:04, 1 user, load average: 0.05, 0.32, 0.36
Tasks: 185 total, 2 running, 183 sleeping, 0 stopped, 0 zombie
%Cpu(s): 94.4 us, 0.4 sy, 0.0 ni, 5.1 id, 0.0 wa, 0.0 hi, 0.1 si, 0.0 st
KiB Mem : 16251608 total, 165508 free, 10923028 used, 5163072 buff/cache
KiB Swap: 8257532 total, 8255112 free, 2420 used. 4833140 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
28352 wfuser 20 0 12.390g 4.694g 32216 S 377.4 30.3 99:52.41 java
24652 wfuser 20 0 7381252 1.111g 18276 S 1.7 7.2 38:33.11 java
709 root 20 0 305076 6136 4772 S 0.3 0.0 1:10.16 vmtoolsd
11853 wfuser 20 0 5929328 899448 14028 S 0.3 5.5 7:49.28 java
CPU usage
mpstat -P ALL
In the example below, there is no deficit of CPU resources.
[root@app1 ~]# mpstat -P ALL
Linux 3.10.0-693.el7.x86_64 (app1) 05/16/2018 _x86_64_ (4 CPU)
11:13:49 AM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
11:13:49 AM all 1.24 0.00 0.41 0.00 0.00 0.04 0.00 0.00 0.00 98.31
11:13:49 AM 0 1.14 0.00 0.38 0.00 0.00 0.04 0.00 0.00 0.00 98.43
11:13:49 AM 1 1.29 0.00 0.41 0.00 0.00 0.04 0.00 0.00 0.00 98.25
11:13:49 AM 2 1.29 0.00 0.44 0.00 0.00 0.04 0.00 0.00 0.00 98.23
11:13:49 AM 3 1.22 0.00 0.41 0.00 0.00 0.04 0.00 0.00 0.00 98.33
mpstat -P ALL 1
In the example below, a CPU process consuming the Business Process was started, which is indicated by the high %usr value.
[root@app1 ~]# mpstat -P ALL 1
11:30:35 AM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
11:30:36 AM all 0.50 0.00 0.25 0.00 0.00 0.00 0.00 0.00 0.00 99.25
11:30:36 AM 0 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 99.00
11:30:36 AM 1 0.99 0.00 0.99 0.00 0.00 0.00 0.00 0.00 0.00 98.02
11:30:36 AM 2 1.01 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 98.99
11:30:36 AM 3 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 99.00
11:30:36 AM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
11:30:37 AM all 16.08 0.00 0.50 0.00 0.00 0.00 0.00 0.00 0.00 83.42
11:30:37 AM 0 10.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00 89.00
11:30:37 AM 1 13.13 0.00 1.01 0.00 0.00 0.00 0.00 0.00 0.00 85.86
11:30:37 AM 2 12.12 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 87.88
11:30:37 AM 3 28.28 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 71.72
11:30:37 AM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
11:30:38 AM all 94.19 0.00 0.25 0.00 0.00 0.00 0.00 0.00 0.00 5.56
11:30:38 AM 0 94.95 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 5.05
11:30:38 AM 1 94.95 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 5.05
11:30:38 AM 2 93.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00 6.00
11:30:38 AM 3 93.94 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 6.06
To find the threads that consume the CPU inside the process, you can use the combination of the following commands:
top -H -p \<jvm\_pid\>to retrieve top CPU consuming threads.\<jvm\_folder\>/bin/jstack \<jvm\_pid\>to understand what consumers do.
There are tools that provide the same result out of the box:
- https://github.com/aragozin/jvm-tools/blob/master/sjk-core/COMMANDS.md#ttop-command
- https://github.com/patric-r/jvmtop
RAM usage
OS level issues
There are two facts that can indicate insufficient memory for Control Tower:
JVM uses a swap. Usually, it's a bad sign because it is difficult for GC to work with the swap.
Java processes using swap
root@fb1575-prod:~# find /proc -maxdepth 2 -path "/proc/[0-9]*/status" -readable -exec awk -v FS=":" '{process[$1]=$2;sub(/^[ \t]+/,"",process[$1]);} END {if(process["VmSwap"] && process["VmSwap"] != "0 kB") printf "%10s %-30s %20s\n",process["Pid"],process["Name"],process["VmSwap"]}' '{}' \; | grep java 9768 java 66068 kB 9820 java 1091596 kB 84939 java 75976 kB 85000 java 325456 kBLogs contain killer messages:
grep oom /var/log/messages or grep oom /var/log/syslog
GC setting issues
In case there is no obvious RAM issues on the OS level, perform a GC activity analysis.
To analyze RAM overconsumption inside an application, use the Heap Dump.
Also, you can view per-thread allocation in real time using the following tools:
- https://github.com/aragozin/jvm-tools/blob/master/sjk-core/COMMANDS.md#ttop-command
- jvisualvm with a memory sampler
IO usage
To get detailed per-process IO usage data, use the iotop utility. But you can also confirm IO deficit by checking the wa value in the top command :
[ec2-user@app1 ~]$ top
top - 12:26:21 up 1 day, 21:51, 1 user, load average: 0.55, 0.41, 0.24
Tasks: 183 total, 1 running, 182 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.7 us, 0.5 sy, 0.0 ni, 98.8 id, 0.0 wa, 0.0 hi, 0.1 si, 0.0 st
wa values higher than 1% indicate the IO deficit.
You can also get wa using the vmstat command:
[ec2-user@app1 ~]$ vmstat 1
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
3 0 2680 232388 0 4333612 0 0 1 17 42 25 1 0 98 0 0
0 0 2680 232612 0 4333612 0 0 0 0 1993 2844 1 1 99 0 0
Disk usage
df -h
[ec2-user@app1 ~]$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/rhel-root 92G 16G 76G 18% /
devtmpfs 7.8G 0 7.8G 0% /dev
tmpfs 7.8G 0 7.8G 0% /dev/shm
tmpfs 7.8G 401M 7.4G 6% /run
tmpfs 7.8G 0 7.8G 0% /sys/fs/cgroup
/dev/sda1 1014M 178M 837M 18% /boot
tmpfs 1.6G 12K 1.6G 1% /run/user/42
tmpfs 1.6G 0 1.6G 0% /run/user/1000
The usage value close to 100% (Use%) is considered an issue and can lead to blocking of JVM threads.