Variables not being used

Grafana Alerting
1

Hey Guys,

First time poster - so please be gentle. I’ve beat my head against the wall trying to understand why a particular alert won’t fill in the variables.

I have created a pretty basic CPU threshold alert:

threadhold
threadhold2224×1507 259 KB

When the threshold gets crossed, it does successfully generate an alert:

[ var='B0' metric='CPU - XX.XX.XX.XX:9100' labels={instance=XX.XX.XX.XX:9100} value=37.43333333275385 ]

However, when I put use any of the following variables, they do not get picked up when they get delivered to slack:

Variable Tests

  1. {{$values.C0.Value}}
  2. {{humanizePercentage $values.C0.Value}}
  3. {{$values.A0.Value}}
  4. {{$values.B0.Value}}
  5. {{$values.C0.Value}}
  6. {{$values.B0.Value}}
  7. {{ $values.B }}
  8. {{$values.A.Value}}
  9. {{$values.B.Value}}
  10. {{$values.C.Value}}

This is what they look like in Slack:

Screenshot 2022-12-08 at 6.10.21 PM
Screenshot 2022-12-08 at 6.10.21 PM1030×1216 169 KB

(As you can see, I’ve tried every variation of the values variable). I have other alerts in production that do work successfully with these variables.

Can someone please point me to what I’m obviously overlooking.

2

Hi and welcome.

What version of Grafana are you using?
At first glance, I would have expected your query to work like this:

image
image1439×200 13.6 KB

image
image1439×201 10 KB

image
image1439×722 44.9 KB

Can you share the output of what you see when you click Preview alerts?

3

Hi @grant2

Thanks for giving me a hand. I like the cleanup in B. Thanks for the tip.

This is the output:

[ var='B' labels={instance=xx.xx.xx.xx:9100} value=16.7944444446928 ], [ var='C' labels={instance=xx.xx.xx.xx:9100} value=1 ]

Alas, the alerts still do not show the values where the variables are :frowning:

Grafana version is v9.2.1 (34f7baebda)

4

Do these work?

{{ $values.B.Labels.instance }}
{{ $values.B.Value }})
{{ $values.C.Labels.instance }}
{{ $values.C.Value }})
5

No Dice:

Screenshot 2022-12-09 at 12.45.30 PM
Screenshot 2022-12-09 at 12.45.30 PM1114×1148 151 KB

6 
{{ index $values.B.Labels “instance” }}

Does the above make any difference?

7

Nope - it just shows that as regular text in the Slack notification. I also tried email to see if it was unique to our Slack implementation, and it showed the same behavior.

8

Hi! Is this a message template or a template for a custom annotation?

9

I am having the same issue. I’m trying to make add dynamic information into custom alert labels. We are using Grafana 8.1.2. I do have a “Classic condition” operation on the alert which is “B” but I’m trying to pull from “A”. The below all render as plain text in the alert instead of the dynamic data I’m wanting. Any ideas or help would be much appreciated.

{{ $values.A0.Labels }}
{{ $values.A0 }}
{{ $values.A }}
{{ $values.A.Labels.instance }}
{{ $values.A0.Labels.instance }}
10

Hi! If A is the query in your alert then you cannot use $values.A because A is a time series, not a single value. This is also why you either need a Reduce expression or Classic Condition as Grafana can only alert on single values, not time series.

I think in your example you will want to use $values.B0, which is the Classic Condition. The number at the end corresponds to the condition in the Classic Condition.

If it helps the documentation on templating labels and annotations can be found here Templating labels and annotations | Grafana documentation.

11

That makes sense. Thanks!