We have dashboards with multiple alerts per panel. What I mean by that is we have a “graphed” metric, which uses $env so that it’s dynamic according to what the user wants to see. We also have additional hidden metrics which we use for our alerting. An example of this setup might look like:
A my.$env.metric.count
B my.prod.metric.count
C my.int.metric.count
And our alerts would be setup something like:
if avg() of B is ABOVE 5 OR if avg() of C is ABOVE 10
So this works great for now. We just alias our hidden metrics so that they’re descriptive: {service}{prod} My Alerting Thing
We’d like to filter our alert panels to only show specific alerts, based on their metric or something similar. We’re using v5.2-beta1, so we do have the alert panel filtering changes. If there would be some way to query the annotation details, which show the metric, which contain our custom alias… I’m thinking we could somehow use that to filter the alert panel (so we’d have an alert panel for Prod and and alert panel for Int).
I don’t expect this would be integrated into Grafana releases, but it would be a custom solution for our internal deployment. Any ideas for things I should look at would be super appreciative.