Labels are not showing in Grafana alert rules

Hi, I have created alert rules in Grafana 8.4 UI. I am unable to see labels defined at target level in Grafana when we have instance in alert expression (for example: sum by (instance) (rate(node_network_receive_bytes_total[2m])) / 1024 / 1024 > 100).

When i query the expression in explorer, I am able to see the labels after changing expression like below,
sum by (instance,product,business_service) (rate(node_network_receive_bytes_total[2m])) / 1024 / 1024 > 100.
But these labels are not being displayed once the alerts fired in Grafana alerts. Please let me know if anyone knows about this issue and probable fix.
Thanks

Hey,

I currently have the same issue.

Did u find any solutions yet?

Hi, I’m having the same issue:

I have a bunch of datapoints with different _field values but otherwise same tags. When setting up a query for a Dashboard, the legend will show me the different datapoints with the unique _field string and all the other tags being identical. When copying the same query into the alerting query field, the legend will show me the _measurement instead of the _field, which is identical in all the records. So I’m running into an error, in which the different dimensions of my alert are no longer distinguishable.
I will try to come up with a workaround; if I’m successful I’ll write a new answer under this one.

@bendor

Can you post all your queries and expressions, and what you see when you click Preview Alerts?


This is the panel as seen in the alert Page. The main identifier is “snmp”, which is the _measurement field of the datapoints I’m querying and since the _measurement field and all the tags are the same, those two series can not be kept apart.


This is the panel as seen in a Dashboard context. The main identifier in this case is “ABB_Port_13” and “ABB_Port_14” respectively, so those two series can be uniquely identified.


When trying to reduce this query to get a two dimensional alert, the tags of the two are the same. When pressing the “Preview Alert” button, the error message

invalid format of evaluation results for the alert definition : frame cannot uniquely be identified by its labels: has duplicate results with labels {IP-Adresse=xx.xx.xx.xx, Raum=xxx, Standort=xxx, Stockwerk=xx, ZaehlerID=42268, agent_host=xx.xx.xx.xx, name=xxxx, type=ABB xxx}

appears.

It seems like even if the _field field is correctly put as the main identifier, the alert still can’t identify the two streams because it tries to do this via the given tags and not the _field, since the main identifier is being replaced with the name of the Reduce Expression, in this Case “B”.

Original Query:

import "date"

from(bucket: "XXXX")
  |> range(start: date.truncate(t: date.sub(d: 90m, from: v.timeRangeStart), unit: 30m), stop: date.truncate(t: now(), unit: 30m))
  |> filter(fn: (r) => r["_measurement"] == "snmp")
  |> filter(fn: (r) => r["IP-Adresse"] =~ /xx.xx.xx.xx/)
  |> filter(fn: (r) => r["_field"] =~ /ABB_Port_13/ or r["_field"] =~ /ABB_Port_14/)
  |> aggregateWindow(every: 30m, fn: mean, createEmpty: false)
  |> difference(nonNegative: false, columns: ["_value"])
  |> difference(nonNegative: false, columns: ["_value"])
  |> map(fn: (r) => ({ r with _value: r._value / 10.0 }))

@bendor Thanks for the info.

Do you wish to get an alert when one of these values exceeds a value (for example, < 100)? Can you explain what condition(s) you wish to alert on?

Sure!
The original Series is data from a Sensor which adds up impulses. Each impulse refers to a certain amount of power used (0.1 Wh / 1 Wh / 10 Wh). The first difference calculates the power used in the given time interval, the second difference calculates the changes in this power usage. If one of those sensors suddenly stops working, no new impulses will be sent, so the value of the first difference drops from value x to 0 and the second difference will be somewhere around -x.
For the beginning, an alert should be sent if the Min value of the series in the time range falls below a fixed threshold, for example -100.

Try adding this:
image

image

then click Run queries and then Preview alerts and post back here…

Thanks for your help. I did the following:


For demonstration, I used the condition < -50 to see, that the evaluation itself works. The right datastream having a min value of -56 is evaluated as 1, while the min value of the left datastream is -19.6 and the Math expression is evaluated with 0. So far so good.
But the problem lies in the way Grafana handles the data reported by my InfluxDB.


When trying to preview the alert, I get the same error as before (next picture) stating that the evaluation results cannot be uniquely identified due to the fact that Grafana omits the Information of the _field field, which is the distinguishing factor of the different data streams.


Grafana seems to handle the result of Flux queries differently, depending of the context (alerting or Dashboard), which causes these problems.

I have had the same issue in the past and resolved it by adding a

 |> rename(columns: {_value: "something"})

Can you try the following? It may require some further iterations to get it working, but you are getting close…

import "date"

from(bucket: "XXXX")
  |> range(start: date.truncate(t: date.sub(d: 90m, from: v.timeRangeStart), unit: 30m), stop: date.truncate(t: now(), unit: 30m))
  |> filter(fn: (r) => r["_measurement"] == "snmp")
  |> filter(fn: (r) => r["IP-Adresse"] =~ /xx.xx.xx.xx/)
  |> filter(fn: (r) => r["_field"] =~ /ABB_Port_13/ or r["_field"] =~ /ABB_Port_14/)
  |> aggregateWindow(every: 30m, fn: mean, createEmpty: false)
  |> difference(nonNegative: false, columns: ["_value"])
  |> difference(nonNegative: false, columns: ["_value"]) <--- Is this row needed ??
  |> map(fn: (r) => ({ r with _value: r._value / 10.0 }))
  |> rename(columns: {_value: "something"})

[/quote]

1 Like

Wow, that worked!
All the names beginning with an underscore are now tags except for _measurement. I’m still wondering why grafana behaves like that; I tried to do a

  |> map(fn: (r) => ({ r with new_tag: r._field }))

which does nothing more than add a new column named new_tag with the same value as the _field field. But in Grafana, this new column was interpreted as a whole new datastream and the original data was no longer named after the _field column but simply _value with all the underscore names as tags. It seems like Grafana queries the existing tags in the database before sending the actual query and just fills up the pre-fetched structure with the data its receives.

Do you know if there is a documentation for this process? It seems like I need a deeper understanding of how Grafana queries Flux to fully use the potential of Grafana.

Awesome, glad it worked!

This was how I figured it out. I have not followed the thread since, but at least in 2021, this was the workaround…

1 Like

Thank you very much, the linked discussion helped understanding how Grafana processes Flux Data.

Thanks also for your help with my original Problem, I appreciate it!