Migrating from Legacy Alerts to New Alerting – Multi-Query Logic Issue

Grafana
1

Hi,
I’m currently using Grafana legacy alerts and planning to migrate to the new alerting system.

The issue I’m facing is with alerts that consist of multiple queries combined using logical operations. After migration, these are converted into classic conditions. Since the new alerts are unidimensional, they do not include the query labels in the evaluation.

I’m considering splitting each query into its own alert rule. Does that approach make sense, or is there a better solution for handling multi-query logic in the new alerting system?

Thanks in advance!

1 Like
2

It depends on what kind of data you query by the rule. Multiple queries in multi-demensional alerts still works but it puts quite strong requirements on the labels of each dimension. If you can achieve that two dimensions from different queries have the same labels (or one is subset of another) then you can do Math expressions on them.
For example

RefID A: {cluster=A, host=B, label=C} 1
RefID B: {cluster=A, host=B } 2

Then Math expression like $A > 0 || $B > 1 would work.

Also, in Grafana 12 we introduced SQL expressions. You can play with it and see if you can convert your rule to SQL expression (it’s not GA yet)

1 Like
4

@yuriy.tseretyan Thanks again for your input earlier! Just wanted to follow up — I’m on Grafana 10.4.2 and trying to migrate, but since my queries have different labels, it seems I’ll need to split them into separate alert rules.

Let me know if there’s any workaround for this setup or if aligning labels is worth considering in my case.

5

Generally speaking, yes. Classic condition does not go away any time soon. If the only problem is to extract information from alerts you still can do that via templating mechanisms Template annotations and labels | Grafana Cloud documentation. For classic conditions there is a special templating approach to access value of each dimension, for example $values.B0.Value where B0, B1 etc is dimension of RefID B (RefID=C is classic condition)

1 Like
6

@yuriy.tseretyan

Here’s an example use case from our legacy alerts:

We have multiple Kubernetes clusters, each connected via a different data source in Grafana.

We use a panel with two queries (one per data source):

  • A: kube_pod_status_phase{job=“kube-state-metrics”, phase!=“Running”, namespace=“example1”} == 1
  • B: kube_pod_status_phase{job=“kube-state-metrics”, phase!=“Running”, namespace=~“example2”} == 1

The alert condition is: if A OR B

If I understood your previous response correctly, I can split this into two separate alert rules.

Alternatively, you mentioned using templating — but in that case, how can I know how many dimensions there will be?

Using loops (e.g., in the labels or annotations) might make the output unreadable or hard to manage.

Is there a recommended way to handle this kind of scenario in the new alerting system?