I have a need to set up Alerts for a number of systems we need to monitor, but the inability to create an Alert on variable based Dashboards is a killer. The only way around this that I know of is to create separate “alert only” dashboards for each metric/system we need to monitor. This becomes quite a challenge as the number of systems and metrics grows. For example, if I have a Dashboard with a variable/drop down to select from 10 different systems, and another variable/drop down to select two different metrics, it would mean I would need to create 20 unique “alert” Dashboards to see if a threshold has been reached on any system, for either of the two metrics being monitored by that dashboard. The feasibility of using the Alert mechanism quickly becomes not feasible when the number of systems or metrics reaches point, exponentially increase with each one being added.
Right now I am being asked to set up a monitoring solution for 30 systems, with approx 20 metrics on each. The sheer number of alert only Dashboards I would need to create is making me think i need with another monitoring solution. I would much rather work with Grafana as a single source for both Dashboards and Alerting, so any news/possible solutions on using variable based query alerting would be awesome.