Alert if any host is above the threshold

Grafana Alerting
1

I am using the following query for alert which should fire if any host is affected (root FS above 94%):

from(bucket: "telegraf")
  |> range(start: v.timeRangeStart, stop:v.timeRangeStop)
  |> filter(fn: (r) =>
    r._measurement == "disk" and
    r._field == "used_percent" and
    r.path == "/" 
  )

Below is my alert configuration and a preview of the data query. My problem is that this alert is firing and then silencing itself because some values are above, some below the threshold.
I am still pretty new to flux and I guess that is somehow picks on of the values. How can I tell Grafana to fire if “any” of the hosts is above the threshold?

Also I could not find any way to extract the “host” field for a short alert message, so any hints where I can find info about it are welcome. I look over the docs of Grafana 9 but the alert section seems very incomplete.

Screenshot 2022-07-25 at 20.12.41
Screenshot 2022-07-25 at 20.12.411433Ă—747 76.6 KB

2

Instead of having query B as a Classic Condition, try changing it to Reduce, then create query C with your expression (>100). Your query is nearly the same as mine. See here and please report back here if it fixes your issue.

3

Sorry, I don’t understand how that is supposed to work. What does reduce do in this case?

I only see a single value selected, which seems random again, so it’s the same as the case before:

Screenshot 2022-07-26 at 00.58.25
Screenshot 2022-07-26 at 00.58.251456Ă—711 81.2 KB

Adding a condition C on that will not make any difference :confused:

Maybe I misunderstood?

4

Creating the Reduce expression in B reduces the query to just the last (latest) values of your disks. This is what Grafana calls a multidimensional alert.

Expression C is where you write it as $C > 94

You then Set alert condition on expression C.

Try it out and see if it works. Also change this to Stat in expression B and expression C so you can see what is being picked up after you run the queries.
image

1 Like
5

OK, I set Expression C to “math” and $B > 94. I see that now all values which are above 94 are converted to 1 and everything else to 0.

I will now try if the alarm works, thanks so far!