"Alerting" on view rule screen, but "Pending" on alert list panel --- why?

I have created an alert and it seems to work. It says Alerting on this screen:

but Pending on this screen (even after 30+ minutes, even though my alerting “for” condition is set for 5 minutes

In case you are wondering, the above panel is this one:
image

It also shows Pending here:

Exploring my Influx query (which is the basis for this alert), I can see “gaps” in the data. There is, however at least 1 reading collected every minute.

Flux query:

from(bucket: "AMPdata")
  |> range(start: v.timeRangeStart, stop: v.timeRangeStop)
  |> filter(fn: (r) => r["_measurement"] == "GeneratorZoneData")
  |> filter(fn: (r) => r["EquipNumber"] == "301" or r["EquipNumber"] == "302" or r["EquipNumber"] == "303" or r["EquipNumber"] == "304" or r["EquipNumber"] == "305")
  |> filter(fn: (r) => r["_field"] == "DewPoint")
  |> filter(fn: (r) => r["MeasType"] == "actual" or r["MeasType"] == "setpoint")
  |> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)
  |> pivot(rowKey:["_time"], columnKey: ["MeasType"], valueColumn: "_value")
  |> map(fn: (r) => ({ r with difference: (r.setpoint - r.actual)}))
  |> drop(columns: ["actual", "setpoint"])
  |> yield(name: "difference")

And one more possible bit of useful info?

image

Also, I edited my Flux query to eliminate the blank values, but Pending still appears (instead of Alerting).

Hello! Thanks for providing all this information and including screenshots!

You are correct in that the evaluation interval is 1 minute and the for condition is set to 5 minutes. In the screenshots I can see that in one example the alert rule has been pending 1min 18s, and in another example the alert rule has been pending for 21 seconds. The alert rule must be pending for the full 5 minutes before it can change to alerting.

It is possible that the missing values are causing the alert to return to either OK state, or DatasourceNoData state, both of which will “reset” the pending timer.

I would recommend changing the alert rule so it queries data over 1 minute intervals instead of 30 second intervals as I understand data is collected once per minute. I might even recommend using max reduce function in a reduce expression and to increase the time range to 2 minutes.

1 Like

@georgerobinson

Thank you for your reply. In your last paragraph, you wrote:

I might even recommend using max reduce function in a reduce expression and to increase the time range to 2 minutes.

Can you explain what you mean by these two instructions?

Hi! If data is collected once per minute, and the alert rule is also set to evaluate once per minute, we want to make sure that there is never an evaluation of the alert rule such that the next data point is not available.

For example, let’s suppose that Grafana runs the first evaluation of the alert rule at 10:00:00. The next data point is due to be collected at 10:00:59, but is not collected until 10:01:04, 5 seconds later than expected. Meanwhile, at 10:01:00 Grafana runs the second evaluation and sees there are no data points between 10:00:00 and 10:01:00.

It is good practice to ensure that there are at least 2 data points for each evaluation so either missed or late data points don’t result in no data. To do this we can either decrease the time between data points (30 seconds rather than 1 minute) or increase the time range to find the maximum value from the last 2 minutes of data each minute.