Alerts firing well after the condition is no longer being met

Grafana Alerting
1

Hi All,

This is my first post and I am new to grafana alerting.
I recently created an alert that I want to have trigger when one of our WAF Rate Limiting Rules triggers.
The query I am using as the alert condition is:
sum(count_over_time({source="<WAF PROVIDER HERE>"}|= “RuleID”:“” [10m]))
I have removed condition B from the alert and have condition C threshold set to “Input A” (the query) “IS ABOVE 1”.
My intent is that I want this rule to fire as soon as possible hence the low threshold.

Yesterday from 15:00 to around 16:27 I sent test requests to our application that would intentionally trigger the WAF Rate Limiting Rule and as expected the alert triggered and posted to one of our Slack channels.

Last night at 20:43 an alert was posted in the Slack channel. I checked our WAF and there were no instances of the rate limiting rule triggering from 20:00 to 21:00 (well outside of the time I had finished testing) and the only instances of the rate limiting rule triggering over a 24h period were during the times I was testing.

When the alert was posted to our Slack channel it showed a couple of things that seemed odd:

Value: [no value] and alertname = DatasourceNoData

I do not understand either why either of these values are present in the alert since the notification policy I am using has the exact same Label and Value that is part of the alert rule itself.

So my questions are:

  • Based on the information provided can anyone explane to me why this alert is firing when the condition is no longer being met?
  • Can someone explaine to me why “no value” and “DatasourceNoData” are included in the notifications that are being posted to Slack?

Thank you

2

I guess because you configured that (or very likely ignored that nodata config, so it is not customized for your need):

3

I believe this did the trick. In my case I set “Alert state if no data or all values are null” to “OK” and “Alert state if execution error or timeout” to “error” re-ran my tests and now the alert is firing in the manner I would have expected.

I appreciate the help @jangaraj, thank you!

1 Like