I’m trying to set up some basic monitoring alerting rules for our systems.
There are several instances of node_explorer being aggregated by Prometheus and further showed by Grafana.
I want to have alerts for, for example, free disk space getting low.
When I set up alerting rules in Prometheus - I have correct alert labels - instance of the job, device, mountpoint. But there are no notifications being sent by Grafana for these rules.
When I set these same rules as a newly-created Grafana-managed alerts - I get notifications all right, but the labels are not processed, they either show up as blank spaces or as non-validated expressions ({{$labels.instance}} and such.
Just to verify: are you using Grafana 8 with the Next Gen Alerting (ngalert) feature toggle enabled? Passing in labels to alerts dynamically inside handlebars will only work with that setup right now…
The Prometheus rule resolves the labels correctly, but does not send alerts.
When I make a parallel rule in a Grafana-managed alert:
node_filesystem_avail_bytes{fstype=~“ext4|xfs”} / node_filesystem_size_bytes{fstype=~“ext4|xfs”} * 100
I am having the same question. I am trying to set up alerting using the new ngalert feature, but I cannot find any useful way to tag my alerts with relevant information about which node is causing the alert.
Is this something being worked on for coming updates? I understand it is not a polished feature yet, hence why I am checking in.
The classic condition type does not have labels available for message templating. You’ll want to create a multi-dimensional rule, which you can find some documentation for here: Create Grafana managed alert rule | Grafana Labs
Hi @davidparrott
If you do not know what to say, it is better to be silent. What is written in the documentation - does not work! If you look at the open problems on GitHub, then you will see that hundreds of people write - labels do not work.
@uanton I definitely keep tabs on GitHub issues etc. I know that Grafana 8 alerting is a significant change from the prior alerting experience and we’ve introduced a lot of complexity. If you have some specific questions or problems please post them.
I understand it is not a polished feature yet, hence why I am checking in.