No data from math expression which should join two sets of series on label subset

Hello, I’m trying to create a somewhat non-trivial alert which takes into account two different prometheus metrics. I have two metric queries A and B, and I’ve added further expressions to threshold them down to essentially booleans: 1 or 0.

When I preview the alert queries, each expression box shows what I expect it to with the exception of one last box that I’m trying to use as the alert condition, which is a Math expression which accepts the last two threshold expression results with $C && $D. I’m expecting it to join based on the labels, but instead it previews as “No data”.

In the other expressions for C I have:

  • {name="foo", event_id="abc"} 0
  • {name="bar", event_id="abc"} 0
  • {name="foo", event_id="def"} 1
  • {name="bar", event_id="def"} 0

For D, I have:

  • {event_id="abc"} 1
  • {event_id="def"} 0

And so for the output to my math expression, I’d expect it to join on the label subset:

  • {name="foo", event_id="abc"} 0
  • {name="bar", event_id="abc"} 0
  • {name="foo", event_id="def"} 0
  • {name="bar", event_id="def"} 0

Instead it just shows “No data”.

Reading through some of the docs here (Queries and conditions | Grafana documentation), it seems that joining will only happen on an exact match of label? Is there any way to achieve this join on the label subset?

The joining in your case should work as expected.

The Math docs includes further details about distinct join scenarios. For example, since the D labels are a subset of C, the following rule applies:

If labels are a subset of the other, for example and item in $A is labeled {host=A,dc=MIA} and item in $B is labeled {host=A} they will join.

Here’s a screenshot of this example using the TestData data source: