Any plans on adding variable query based alerts?

I have a need to set up Alerts for a number of systems we need to monitor, but the inability to create an Alert on variable based Dashboards is a killer. The only way around this that I know of is to create separate “alert only” dashboards for each metric/system we need to monitor. This becomes quite a challenge as the number of systems and metrics grows. For example, if I have a Dashboard with a variable/drop down to select from 10 different systems, and another variable/drop down to select two different metrics, it would mean I would need to create 20 unique “alert” Dashboards to see if a threshold has been reached on any system, for either of the two metrics being monitored by that dashboard. The feasibility of using the Alert mechanism quickly becomes not feasible when the number of systems or metrics reaches point, exponentially increase with each one being added.

Right now I am being asked to set up a monitoring solution for 30 systems, with approx 20 metrics on each. The sheer number of alert only Dashboards I would need to create is making me think i need with another monitoring solution. I would much rather work with Grafana as a single source for both Dashboards and Alerting, so any news/possible solutions on using variable based query alerting would be awesome.

1 Like

This is a tricky problem, and there’s not yet a consensus for how this should work. Alert rules needs to be evaluated on the Grafana server, and variables are only available in the browser at this point.

Right now, alerting is a part of the graph panel, which is something we’d like to move away from. If alerting ends up existing outside dashboards, what variable should be used?

If you have any ideas on this, the team would be interested in hearing your thoughts!

We ended up scripting a Python tool to build ‘alert dashboards’ (by posting them to the grafana API) that are not meant for human consumption but, instead just have a panel for each thing we want to alert on.
At the moment it’s the only way I know of to use Grafana for alerts w/out being buried in GUI clickfest.