Would be curious as to the priority being placed on this considering that every competitor to Grafana allows for this capability. My main fear is that it will make it’s way into their paid offering before it makes its way into the Community version.
Has there been any more discussion or progress surrounding this? I know that you suggested it may not be what people want but by the looks of it, it’s what people need and it shouldn’t be too difficult to implement as long as there are some guardrails.
1 Like
We have around 500+ servers with some more than 1TB RAM.
We have prometheus node exporter running on those servers.
We need email alerts on
- cpu usage high
- ram usage high
- network usage high
- disk usage high
- root fs more than certain % (80)
- sys load > x %
- system down
Multiply this 7 * 500 servers. How is this even feasible to sit there and create a separate dashboard for 500+ servers? How is this efficient to create 3,500 alerts for by going to each server’s dashboard and editing panel and creating alert?
4 Likes
You can export it as a JSON, parse it with a python script or whatever language you want, and just generate the same dashboard 500 times but with different server names. Alerting rules and every setting will remain the same. Even the requests for importing can be done in the same script.
Exactly. How in the world is this not a priority. With the scalability of Prometheus you would think this would be high on the list, if not already implemented. Template variables in dashboards are what allows us to graph data like this in the first place. I’m not making 100 duplicate dashboards just so I can have alerts.
1 Like
Hi guys. Any news on this topic?
So any updates on this very crucial thing?
1 Like
Hi,
Any workaround ?
I’d like to get an answer on this as well. I see plenty of people complaining about this issue for years and yet nothing has been done. I’m not concerned about “why” template variables are not usable in alerts, I just want them to work.
Meaning no disrespect, but if Grafana Labs wants to crow about how great template variables are(and they are), then why in the world are they not supported in alerts?
At present I have close to 20+ alerts that fire off based on a variety of metrics. I can easily create queries with template variables but when when it comes to alert queries I can’t use them at all! Template variables are not a “nice to have” in Grafana, they are integral to setting up dashboards that are concise and consistent. It’s what makes Grafana Grafana! But you can’t use them with alerts? Asking us to do “workarounds” is not the answer. Alerts MUST be able to support template variables just like you can anywhere else in Grafana. I can’t even use a simple constant variable for the alert message as they are not interpolated in the alerts.
As Grafana Labs expands its offerings to the cloud (either via their own SaS offering or through Amazon’s Amazon Managed Grafana Service(AMG)) the most important aspect of usability, using variables in alerts is receiving little to no serious attention. I don’t know if Grafana Labs realizes what an impediment this is to its customers.
This is a huge missed opportunity for Grafana Labs so much so, that I’m looking for alternatives to Grafana (thought I’d never say that).
Should we open up a bug for this issue or make a feature request?
6 Likes
I’ve added a feature request to allow using template variables in alert queries:
2 Likes
Hi,
You created a discussion not a pull request.
Correct your request.
Why would I create a PR? This is a feature request. I have no code that needs to be merged.
why not using the ngalert … this would solve your problem
ngalert is probably great for brand new dashboards but it’s not so great with existing dashboards. When you turn this on feature Grafana will attempt to convert your existing alerts to make them work with the ngalert engine. I expect there there is a vast number of Grafana users who are not using this engine yet and have dashboard they’ve invested many hours to create. You can’t just ask them to “use ngalert” because making that switch is not insignificant. There’s a reason why ngalerts is a feature toggle and has not replaced the existing engine out of the box. So, let’s table the idea of ngalerts and focus on the fact that template variables do not work with the current alerting engine.
I think there are severe limitations with template variables and alert queries. In my mind the core strength of Grafana is its templating feature. From an investment point of view, the “templating engine” should be the main focus. Yet, when I create an alert I have to stop and think “Oh, I’m using an alert. I can’t use template variables with alert queries”. I still have not gotten a good reason why they can’t be used with alerts and this problem has existed for many, many years. Grafana Labs keeps “kicking the can down the road” and doesn’t fix this issue. Why, I don’t know.
When newbies run into this issue they will inevitably jump onto the Community forum or Git and ask why they can’t use template variables with alerts and the entire discussion is rehashed. There are no real solutions just (in my mind), just hacks. So a lot of time being wasted on not getting this fixed without switching to a new alerting engine.
The fact that they can be use in alert query is giving Grafana a black eye. The most powerful and fundamental aspect of Grafana is its templating and so is alerting. That those two core aspects of Grafana can’t be used together without a hack is a BIG problem.
I don’t understand why Grafana Labs can’t make a tweak to the EXISTING alerting engine to allow it to interpolate template variables. I realize they’ve got a lot of work on their hands but since they are expanding their user base via SaaS offerings (even AWS now has a native Grafana service). The calls for making the template variables work within alert is just going to increase.
6 Likes
Updating this thread to include how we’ve resolved this with Grafana Alerting available in 8.x and newer.
Template variables are still not supported, however, with multi-dimensional alerts you can alert on each instance of a series returned from a query.
Using the labels from the query, you can include information for each alert in your notification.