I am using the following query for alert which should fire if any host is affected (root FS above 94%):
|> range(start: v.timeRangeStart, stop:v.timeRangeStop)
|> filter(fn: (r) =>
r._measurement == "disk" and
r._field == "used_percent" and
r.path == "/"
Below is my alert configuration and a preview of the data query. My problem is that this alert is firing and then silencing itself because some values are above, some below the threshold.
I am still pretty new to
flux and I guess that is somehow picks on of the values. How can I tell Grafana to fire if “any” of the hosts is above the threshold?
Also I could not find any way to extract the “host” field for a short alert message, so any hints where I can find info about it are welcome. I look over the docs of Grafana 9 but the alert section seems very incomplete.
Instead of having query B as a Classic Condition, try changing it to Reduce, then create query C with your expression (>100). Your query is nearly the same as mine. See here and please report back here if it fixes your issue.
Sorry, I don’t understand how that is supposed to work. What does reduce do in this case?
I only see a single value selected, which seems random again, so it’s the same as the case before:
Adding a condition C on that will not make any difference
Maybe I misunderstood?
Creating the Reduce expression in B reduces the query to just the last (latest) values of your disks. This is what Grafana calls a multidimensional alert.
Expression C is where you write it as $C > 94
You then Set alert condition on expression C.
Try it out and see if it works. Also change this to Stat in expression B and expression C so you can see what is being picked up after you run the queries.
OK, I set
Expression C to “math” and
$B > 94. I see that now all values which are above 94 are converted to 1 and everything else to 0.
I will now try if the alarm works, thanks so far!