How to create alert based on multiple queries?

[hywak]

For few days now I’m trying to create alert when ratio produced to consumed messages is not equal to 1. I don’t want to do it based on prometheus due to each time service is restarted, counter is reseted also I don’t want to create gauge because I’d have to store count of produced messages somewhere in db. I decided to use logs stored in elasticsearch. Here the problem is that service which produces messages is logging to one elasticsearch and service which consumes them to another elastic. According to my knowledge it is impossible to create alret based on 2 queries.

I’m out of ideas how to approach this in other way around. Maybe I’m missing something or there is other way how to create alert like this based on 2 data sources?

[fadjar340]

You can use --Mixed-- data source then you can choose elasticsearch data source that you need, before it, build two elasticsearch data source for each index.
You can create multiple metric using mixed data source.

Other way,
First, choose the metricbeat, for example, then build the metric,
Second change the data source for heartbeat, for instance, then Add Query, choose the suitable datasource, then you can show in the single graph, multiple datasource.

Then, you can set the alert for each metric with specific conditions…

Regards,
Fadjar Tandabawana

[nikolaosstrongioglou]

I have the same issue but I cannot really undestand @fadjar340 reply. Could you be more specific? I cannot follow your train of thought.

[fadjar340]

Sorry for my bad English, not my mother tongue…

Oke, I’ll explain

1. Create the different datasource, in my case, I use metricbeat and auditbeat
2. Build new dashboard using Graph Visualization
3. Select the datasource to --Mixed–
   
4. Add first query A for first datasource by selecting the datasource using +Query, select Metricbeat
   
5. Build the query as you need…
6. Add second query B for datasource auditbeat using select the datasource using +Query
   
   
   
   Above steps, create multiple query in the same Graph
   
   

Now, we go to the Alerts
7. Open the Alert tab and create the alert using the Create Alert Button
8. Create the Alert for Query A

9. Add the alert for query B using sign +
10.You can select the condition of the alerts, OR - AND

11. Change the second alert to query B

12. Now you have multiple alerts in the same Graph with multiple datasource.

13. SAve the dashbord first.
14. You can test the alert using Test Rule button in the last of the configuration
15. Below the test result

I wish this helps you…

[nikolaosstrongioglou]

Thanks a lot for your detailed reply. Unfortunately, this does not help me since I want to do math between these two queries and not just check boolean conditions. I sincerely appreciate your reply though!

[rakopoul]

I want to do something similar. I have the following problem:

I want an alert looking like this:

It is essentially firing when disk usage is above 75% AND also free space is less than 1TB. But when one of the hosts meets both conditions it seems that Grafana does not relate the two queries and send the alert for the host that matches both conditions. Instead it is sending me in the email all the hosts that match any of the two conditions.

Can you help on this one?

[grant2]

@rakopoul

What version of Grafana are you using and what is your datasource?

[rakopoul]

Hello,

We are using Grafana 8.4.7 with legacy alerting still enabled (planning to migrate soon to unified alerting).
Our data source is influxdb.

Thanks in advance

[grant2]

Your alert should fire only when both conditions are met, but apparently it is not. Maybe it’s a bug or something that has been replaced with the new alerting.

If you use (or can upgrade to) Flux, then you can refer to Example 2 in this tutorial that alerts on 2 conditions. See Example 2.

[rakopoul]

Thanks i managed to create the two expressions and i see that final math expression works as expected. Though it seems that i can not create an alert based on an expressions (seems not supported with legacy alerting). Can you confirm this?

[grant2]

I have not used Legacy alerting in years, so I cannot answer your question. Sorry.

[rakopoul]

I tried to migrate in my test environment to unified alerting and it works fine. So it is up to us to migrate our system to latest unified alerting.

Thanks a lot for your help.