I had a lag in metrics being displayed in my dashboards. This lag increased from 3 mins to 14 mins over the weekend.
The memory on my box was at 95% so I increased the memory from 8GB to 16GB.
Machine spec
OS Version Ubuntu 16.04 LTS
Release 16.04 (xenial)
Kernel Version 4.4.0-103-generic
Docker Version 17.09.0-ce
CPU 4 cores
Memory 16 GB
Memory reservation is unlimited
Storage
Driver → overlay2
Backing Filesystem → extfs
Supports d_type → true
Native Overlay Diff → true
Memory swap limit → 2.00GB
The lag is now gone but I have noticed a serious decrease in metrics received.
(carbon.agents.*.metricsReceived)
The outage around 10:30 is for machine reboot after memory was increased.
Can someone explain why this is the case?