Prometheus node_explorer labels not showing up in Grafana Alert description template

I’m trying to set up some basic monitoring alerting rules for our systems.
There are several instances of node_explorer being aggregated by Prometheus and further showed by Grafana.
I want to have alerts for, for example, free disk space getting low.
When I set up alerting rules in Prometheus - I have correct alert labels - instance of the job, device, mountpoint. But there are no notifications being sent by Grafana for these rules.
When I set these same rules as a newly-created Grafana-managed alerts - I get notifications all right, but the labels are not processed, they either show up as blank spaces or as non-validated expressions ({{$labels.instance}} and such.

What am I missing here?

Hi @chiffa

Just to verify: are you using Grafana 8 with the Next Gen Alerting (ngalert) feature toggle enabled? Passing in labels to alerts dynamically inside handlebars will only work with that setup right now…

Hi, @mattabrams
Yes, I’m using Grafana 8 with ngalert feature…

is $labels.instance defined as a template variable in a dashboard, and is it used in the query that you are alerting on?

Well, I tried creating the alert from scratch, building on a Prometheus-based alert rule…
Like this:

The Prometheus rule resolves the labels correctly, but does not send alerts.

When I make a parallel rule in a Grafana-managed alert:
node_filesystem_avail_bytes{fstype=~“ext4|xfs”} / node_filesystem_size_bytes{fstype=~“ext4|xfs”} * 100

It sends alerts as described in notification channels, but does not resolve labels



I am having the same question. I am trying to set up alerting using the new ngalert feature, but I cannot find any useful way to tag my alerts with relevant information about which node is causing the alert.

Is this something being worked on for coming updates? :slight_smile: I understand it is not a polished feature yet, hence why I am checking in.

The classic condition type does not have labels available for message templating. You’ll want to create a multi-dimensional rule, which you can find some documentation for here: Create Grafana managed alert rule | Grafana Labs

1 Like

Hi @davidparrott
If you do not know what to say, it is better to be silent. What is written in the documentation - does not work! If you look at the open problems on GitHub, then you will see that hundreds of people write - labels do not work.

@uanton I definitely keep tabs on GitHub issues etc. I know that Grafana 8 alerting is a significant change from the prior alerting experience and we’ve introduced a lot of complexity. If you have some specific questions or problems please post them.

1 Like