Alerting on peaks

ayushgoyal1803 · September 26, 2020, 4:07pm

Hi,

I am creating several panels in dashboard and each one of them is count based on conditions. I want to create alerts when sudden spike appears. How can i achieve that? I tried below methods.

I calculate average and comparing number of records receiving but this is not fulfilling our requirement.

Now, I am thinking of using percent_diff().

I am using grafana 6.2.5

fadjar340 · September 27, 2020, 10:28am

Try to use max value

ayushgoyal1803 · September 28, 2020, 12:59pm

max value can be anything and range can also increase. I am looking for low width peaks which will generate due to system failures.

fadjar340 · September 28, 2020, 1:14pm

I think you can use derivative…
For sure, Grafana7.x have this, I also use this derivative value to get the diff of the values.

ayushgoyal1803 · September 28, 2020, 1:19pm

I can’t find proper documentation where usage and details are mentioned.

ayushgoyal1803 · September 28, 2020, 2:30pm

Hi @fadjar340,

As you suggest, I can see derivate is the thing i was looking for but how to apply alert on derivate of count instead of count?

Thanks

fadjar340 · September 28, 2020, 2:52pm

Check my post here
To visualize the derivative only, you can hide the main metric but the derivative metric

ayushgoyal1803 · September 28, 2020, 3:09pm

Currently, I am having metrics and graph like this and i want to apply alert on yellow. How to do that?

As you can see here i have only Query A present here.

fadjar340 · September 30, 2020, 4:49am

Just select the eye picture beside Metric Count, then you will have only Derivative metrics show, then build the alert