Dynamic alerting thresholds in unified alerting

Dynamic alert thresholds could be achieved by joining the time-series query with data from another source (like a CSV or SQL table). That would produce some extra columns with thresholds that could be referred to in an alert expression.

I tried to do this in Grafana unified alerting but I found that it is basically impossible due to restrictions on the format of the data that is produced by the query (needs to be wide-format time-series data).
There are some exceptions mentioned here: Alerting on numeric data | Grafana documentation
But basically it seems impossible to do.

Next I will try to get this to work in Influx-DB alerting. Does anybody have a better idea?

Now this post has been unanswered for 21d now so I should probably add some info:

What do I mean by “dynamic alerting thresholds”?
In Grafana unified alerting you can set a threshold like: If value > 80 then alert
Grafana users can modify the threshold (80 in this case) to another value. So, in that sense the threshold is dynamic.
However, what I want is that the value “80” is read from another datasource like so:
If value > thresholdFromDatasource then alert.

I actually found some interesting info in This webinar video. 23 mins into the video they show a method that introduces a dynamic threshold from another datasource. I did not try this yet, but I’m afraid I may run into trouble when using Influx-DB instead of MySQL queries.

1 Like

Turns out I needed to reorganize the schema of my InfluxDB bucket. Instead of having one measurement per “channel” I now have one measurement with one tag per “channel” of measured data. I’m still struggling with the math function that combines all the queries, but I think I will manage…

Just to join in on your lonely thread, where did you learn that you need to:

have one measurement with one tag per “channel” of measured data.

? I am curious to see your InfluxDB query and resulting alerts.

I am not using Dynamic Alerting Thresholds, but have struggled with Multi Dimensional Alerts. Am planning to do a writeup soon of my workaround and will post here.