Alerting on multiple series with different threshold for each series : alert definition error

  • What Grafana version and what operating system are you using?

Grafana 8.1.2
Debian Buster

  • What are you trying to achieve?

I’m trying to set threshold alerts on multiple series.

The thing is the threshold differs depending on the series, as they can be temperatures, current measurements, etc. So I can’t just compare all series to the same constant min and max.

Each series has its threshold:

ID_1: [0; 100]
ID_2: [-1000; 1000]

There may even be series with no min or no max.

More on the use case in this GH comment.

  • How are you trying to achieve it?

I don’t know of any way to inject the threshold for each series into the equation in the alert framework. I’m open to suggestions.

I think I found some sort of workaround: I created a dedicated measurement in which I wrote for each series ID the tagset of the series (in fact, “ID” tag would be enough as it is unique) and threshold values “min” and “max”, for an arbitrary date (January 1st 2020).

The plan is to

A: query my series on last 3h groups by ID
B: reduce series to min
C: reduce series to max
D: query the min thresholds around 2020-01-01 from dedicated measurement
E: query the max thresholds around 2020-01-01 from dedicated measurement
F: compare B to D and C to E and alert for each ID if value out of bounds

A: SELECT "value" FROM "measurement" WHERE $timeFilter GROUP BY "id"
B: Operation Reduce Function Min Input A
C: Operation Reduce Function Max Input A
D: SELECT "min" FROM "metadata" WHERE $timeFilter GROUP BY "id"
E: SELECT "max" FROM "metadata" WHERE $timeFilter GROUP BY "id"
F: math: !(($B < $D) || ($C > $E))

(The first issue is that queries timeframes in the alert framework have to be relative to now(). I set an arbitrary long period to be sure it still works in several years. Still, it would be nice if we could use an absolute timeframe. I don’t get the reason for this limitation.)

  • What happened?

In the “Preview alerts”, I get

invalid format of evaluation results for the alert definition : frame cannot uniquely be identified by its labels: has duplicate results with labels {}

If I change F not to use common constant values, it works (not what I want to do, but I mean the alert rule works).

F: math: !(($B < 0) || ($C > 1000))

The preview consists of N lines reading:

[ var='F' labels={id=ID_1} value=1 ], [ var='C' labels={id=ID_1} value=0 ], [ var='B' labels={id=ID_1} value=0 ]

  • What did you expect to happen?

I expected the same result when using

F: math: !(($B <  $D) || ($C > $E))

When displaying the Query stats under F in both cases (with D and E or with constants), I get N values like:

F ID_1    F ID_1
1         0

1 is the result or math F meaning value inside bounds. 0 means out of bounds.

So at this stage, I can’t tell the difference in the expression result between comparison to constants thresholds or thresholds from D and E (only the 0/1 values change of course, because the constant thresholds values are incorrect).

My questions are:

  • Why doesn’t my alert work as I expected? Why do I get this error?

  • Is there a better to do what I’m trying to achieve in the first place?

Subsidiary question: I also need to skip the comparison if a threshold is empty (I can have series with no theoretical min or max), how should I manage this special case?

Thanks.

1 Like