Alerting from multiple sources

Hello,

I’m trying to generate an alert for a data source that has lots of variables and because of that I was forced to break it into multiple data sources in the alert

For say of argument I have A,B,C

I tried then

AR = Reduce A on last
BR = Reduce B on last
CR = Reduce C on last
Then J = Classic condition where A last > 0 or B last > 0 or C last > 0

But because J is using “Classic Condition” I cannot extract the labels or the values for my alert description.

I decided then to do instead the following:

AR = Reduce A on last
BR = Reduce B on last
CR = Reduce C on last
M = Math $AR> 0 || $BR> 0|| $CR> 0

But then I get no data for this last aggregation step.

There is a way to aggregate this data sources to generate a single alert or do I need to create similar alert rules for data source A, B and C?

I can provide more details if needed.

Thanks.

2 Likes

Looks like this was asked already, not much luck: [Grafana 8] [Unified Alerting] Single alert on multiple queries using different data sources - #3 by whatsupbros

Here too: prometheus - Grafana 8 - reduced alert on multiple queries - Stack Overflow

Why is this so? What is the message / output that is shown in Grafana when you preview the alert?

It says no data.

José Vicente Nuñez

What is your data source? Can you post all of the queries and expressions used in your alert?

All the datasources are similar to this (I’m using the Zabbix plugin)

And they produce a proper time series.

The I reduce each time series (Like A → REDUCEA):

And then I do math with all the reduced series (Because I do want to be able to extract the labels):

I tried also using a “Classic” expression directly instead of A-> reduce A → Classic (with an or for each series) but then I cannot get the value or the label from the result in the alert.

Thanks.

Everything you showed and explained looks fine to me.

Are you alerting off the correct expression? (MATHALL?)

Apologies, missed that. Yep, looks like it is the case:

Which version of Grafana are you using? I am using v9.2 OSS. I noticed my screenshot says “Set Alert Condition”, while yours says “Define Alert Condition” and you do not have a radio button like I do.

Mine is 8.5.4. Maybe is time to upgrade.

José Vicente Nuñez

Hello,

I did upgrade to Grafana 9.2.x but that didn’t make a difference.

Also, after reading the resampling documentation some more, I found the following bit:

Resample changes the time stamps in each time series to have a consistent time interval. The main use case is so you can resample time series that do not share the same timestamps so math can be performed between them. This can be done by resample each of the two series, and then in a Math operation referencing the resampled variables.

So my expression turned to this:

$A (original) → $C (resampled to 1m, fill pad) → $E (reduce last)
$B (original) → $D (resampled to 1m, fill pad) → $F (reduce last)

Math $G = $E > 0 || $F > 0, returns no data

Math $E >0 or Math $F > 0 work, but not when you combine the expressions together.

I don’t think Grafana supports this, but please correct me if I’m wrong.

I have this expression working fine in my alerting:

${B}>65.0 ||  ${B}<45.0

Do you perhaps need the { } ?

I’m seeing the exact same behaviour as the poster. Is it then the case that Grafana won’t support multiple data sources combined via a Math expression in this way? Can someone confirm?

(@grant2 you’re using a single refID, ${B}, in your Math op, my understanding is the poster is interested in using different refIDs and specifically such that they originate from different data sources.)

I had this same issue and determined that the labels were different between the two queries - specifically, I had two mimir metrics and was trying to join on this. I fixed it by dropping the name label from each:

label_replace(my_mimir_metric, "__name__", "", "", "")

How to get the name of datasource / Query which generated the alert ?
I want to send the notification with data source name in it.

Welcome @shubjain26

There are a lot of posts covering this, but I recommend you read through these and see what you can come up with.

If you are using Slack or Telegram, these will help:

1 Like