Hi all!
In short, our metrics temporarily spike in our Grafana graphing, triggering our monitoring alert and shortly (~5minutes) corrects its reading to resolve the alert.
As you can see in the image below, we recieve an event that alert occurs on 13:05:00 and then fixes itself at 13:06:15.
There interesting part are the readings. It triggers with value 1227, which is passes our 1100 threshold, yet at our max and current values for each metrics does not go as high as 4.
On another graph, we spotted the temporary (and intermittent) spike, which I suppose is meant to trigger the alert, but then corrects itself. This behaviour isn’t true to our datasource.
I can confirm that the metrics on our datasource (Graphite) is consistently valid, so problem is isolated on Grafana.
So far we think behaviour only exists when we add in alert monitoring.
Is this a known issue? Is anyone able to replicate this? Please let me know what I can provide to help troubleshoot this.
Kind Regards,
Tuan
