I am using Grafana 8.5.0 and have a grafana dashboard which basically shows number of waiting calls. I have grouped them according to group_id. So my query would look something like
sum by (group_id) (increase(call_counter_total{call_type="PHONE", group_id!="UNKNOWN"}[15m])) * on(group_id) sum by (group_id) (agent_status > 0)
Now I want to create alerts whenever this query returns 0 for more than 10 minutes. In the alerts section I went ahead and added a rule
- Static rule name(without any template): Call alerts
- Evaluate
every 1 min For 1 second. - And in condition, I have passed
avg() of query(A,15m,now) is below 0.0001
The issue with this is that when the state is OK, and the condition is hit an alert is triggered, and the state changes to ALERTING. Now when it is in ALERTING state and another group_id or a group of group_ids hit the condition, I am not getting the alerts for these. I was looking around what is causing the issue and came across a few conversations on Github.
I believe my issue is almost similar to this issue. I was also looking at a few issues which discusses it and came across issue 6041 on github which recommends usage of multidimensional alerts based on labels.
I have tried it by basically adding a reduce expression B which uses Mean function. And I have added another expression C to perform the condition. I am not sure what to do next because when I update my alert to use the condition C it throws an error telling, this datasource doesnt support alerting queries. Attached a few screenshots for reference. Also can someone provide help/show me documentation on how to use legends.