Grafana alert Math expression for multiple queries with multi-dimensional data

Grafana Alerting
1

HI guys,
Grafana v10.4.7 (8d068faebe)
My case is to setup an alert to check if my service deployed to different clusters has the same release-id. The metrics are exposed to two data sources so that I have two queries A and B:
A: group(
kube_deployment_labels{namespace=“namespace”, label_component=~“mysvc”,clustername=“cluster1”})
by (clustername, label_component,label_release_id)

A result:
{clustername=“cluster1”, label_component=“mysvc”, label_release_id=“main.b99.aabcd”} 1

B: group(
kube_deployment_labels{namespace=“namespace”, label_component=~“mysvc”,clustername=“cluster2”})
by (clustername, label_component,label_release_id)

B result:
{clustername=“cluster1”, label_component=“mysvc”, label_release_id=“main.b98.32dfd”} 1

I am using Math expression with $A==$B and expected 0 as A has different label_release_id with B, but the final result is 1 and it shows warning:
2 items dropped from union(s): [“$A == $B”: ($A: {clustername=cluster1, label_component=mysvc, label_release_id=main.b99.aabcd}) ($B: {clustername=cluster2, label_component=mysvc, label_release_id=main.b98.32dfd})]

I expect to be able to compare labels comprehensively instead of just comparing aggregate values. Can anyone help? Thanks.

2

From this comment Grafana OSS alerting - #2 by pepecano, I thought it should compare with same label set, but dont know why it failed my case :sweat_smile:

3

Hi,

You’re right - Math expression will compare the same label set - the series you’ve provided are different label sets (the label_release_id is different), the expression does not know how to compare those sets, so it drops it (I’ve once written a blog post somewhat explaining what’s happening - in PromQL but the gist is the same).
The expression $A==$B would compare only values of the same label sets, so that’s why it’s not working. I’m not sure (or like 80% sure) that you can’t do something like that in PromQL.

4

Hey @dawiddebowski , appreciated the help! Do you have any suggestions how I can accomplish my task? Thank you very much.

5

The pain point is I have two data sources so that I think I can not use PromQL join function to compare label_release_id. There are limited tools in Alerting section which dose not like Dashboard where I can transform the data structure. I am trying to use $A + $B, it still only merge the value with same label sets just like an inner join :face_with_tears_of_joy:.