How to include only results that have matching labels in both time series

  • What Grafana version and what operating system are you using?

Grafana 12.2, Windows/Chrome (for client; for servers - some Linux in the cloud)

  • What are you trying to achieve?

I have 2 metrics, one produces let’s say 40 time series for a given label, the other - 100 (and those 40 among them). I want to display max values for each of the time series next to label value in a table, like this:

  • serviceA, max heap size, container limit
  • serviceB, max heap size, container limit
  • and so on
  • How are you trying to achieve it?

in a dashboard I have 2 queries:
memory_limit: max by (container) (container_spec_memory_limit_bytes{k8s_cluster=“prod”})
jvm_memory_used_bytes: max by (container) (jvm_memory_used_bytes{k8s_cluster=“prod”, area=“heap”}); I keep them “hidden”

and 2 expressions, both “Reduce/max” configuration for corresponding query

  • What happened?

so what I have is a table view with a “drop down selector” where I can only choose one value at a time for display, which look like this:

  • memory_limit_max serviceA
  • memory_limit_max serviceB
  • jvm_used_bytes_max serviceA
  • jvm_used_bytes_max serviceB
    etc.
  • What did you expect to happen?

not that I expected something right away, but how do I merge these 2 sets of time series, leaving only those that have matching label value in both; and then display them in one table without having to select anything?

  • Can you copy/paste the configuration(s) that you are having problems with?
{
  "annotations": {
    "list": [
      {
        "builtIn": 1,
        "datasource": {
          "type": "grafana",
          "uid": "-- Grafana --"
        },
        "enable": true,
        "hide": true,
        "iconColor": "rgba(0, 211, 255, 1)",
        "name": "Annotations & Alerts",
        "type": "dashboard"
      }
    ]
  },
  "editable": true,
  "fiscalYearStartMonth": 0,
  "graphTooltip": 0,
  "id": 1090,
  "links": [],
  "panels": [
    {
      "datasource": {
        "default": false,
        "type": "prometheus",
        "uid": "metrics-prod"
      },
      "fieldConfig": {
        "defaults": {
          "custom": {
            "align": "auto",
            "cellOptions": {
              "type": "auto"
            },
            "inspect": false
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              },
              {
                "color": "red",
                "value": 80
              }
            ]
          }
        },
        "overrides": []
      },
      "gridPos": {
        "h": 8,
        "w": 12,
        "x": 0,
        "y": 0
      },
      "id": 1,
      "options": {
        "cellHeight": "sm",
        "footer": {
          "countRows": false,
          "fields": "",
          "reducer": [
            "sum"
          ],
          "show": false
        },
        "showHeader": true
      },
      "pluginVersion": "11.2.2+security-01",
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "metrics-prod"
          },
          "editorMode": "code",
          "expr": "max by (container) (container_spec_memory_limit_bytes{k8s_cluster=\"prod\"})",
          "hide": true,
          "instant": false,
          "legendFormat": "__auto",
          "range": true,
          "refId": "memory_limit"
        },
        {
          "datasource": {
            "name": "Expression",
            "type": "__expr__",
            "uid": "__expr__"
          },
          "expression": "memory_limit",
          "hide": false,
          "reducer": "max",
          "refId": "memory_limit_max",
          "type": "reduce"
        },
        {
          "datasource": {
            "type": "prometheus",
            "uid": "metrics-prod"
          },
          "editorMode": "code",
          "expr": "max by (container) (jvm_memory_used_bytes{k8s_cluster=\"prod\", area=\"heap\"})",
          "hide": true,
          "instant": false,
          "legendFormat": "__auto",
          "range": true,
          "refId": "jvm_memory_used_bytes"
        },
        {
          "datasource": {
            "name": "Expression",
            "type": "__expr__",
            "uid": "__expr__"
          },
          "expression": "jvm_memory_used_bytes",
          "hide": false,
          "reducer": "max",
          "refId": "jvm_memory_used_bytes_max",
          "type": "reduce"
        }
      ],
      "title": "Panel Title",
      "type": "table"
    }
  ],
  "schemaVersion": 39,
  "tags": [],
  "templating": {
    "list": []
  },
  "time": {
    "from": "now-6h",
    "to": "now"
  },
  "timepicker": {},
  "timezone": "Europe/Tallinn",
  "title": "experimental",
  "uid": "be7hy1f17vzeoa",
  "version": 1,
  "weekStart": ""
}
  • Did you receive any errors in the Grafana UI or in related logs? If so, please tell us exactly what they were.

no

  • Did you follow any online instructions? If so, what is the URL?

no

Hi, if you’re open to suggestions, I think I could help (if not, let me know I’ll try harder with what you have :smile:)

  1. First, I wouldn’t use the expressions. PromQL provides aggregations over time, which work like: max_over_time((your_query{})[t1:t2]) (I’ll explain what t1 and t2 are at the end but if you’ve already known that, you can skip that). I’d also use Table format instead of time series and, using the aggregation over time, I can use instant query, so that I would have only one point in time - like this:

  2. I would still have one table per one series returned - that I can fix by going into Transform data tab and in there add transformation Join by field. For Field, pick container (since it’s the common field returned by your queries), as for Mode - decide what you want. You’ve said that one of the query has 60 series that don’t match the other query - if you want to include those, select Outer join (it will leave empty fields in the query that doesn’t match. If you want to display only series that match both queries - pick Inner.


  1. Notice that there are two Time fields, neither of which are really necessary - you can use transformation Organize fields by name to not only hide the time fields but also rename the Value #used names:

That should give you panel like this (I hope I understood you correctly :sweat_smile:):

You can of course change the unit for Limit and Used columns in fields override (if you need help with how to do that too - let me know).

Aggregations over time - a quick explanation

When you create a query in Grafana, you can pick the time range for the query to display - typically Last n hours. Notice that Grafana also has to know where the points are - Prometheus (and coherently Grafana) assumes that the points are in constant intervals, e.g. every 15 seconds. This 15 second window is called an interval. When you create a query you rarely ever think about it, because Grafana has that interval in datasource options (by default it’s 15 seconds). Why is this important?

In aggregations over time you have to pass those two numbers - for how long you’re looking back in time and with what resolution. So queyr max_over_time((your_query{})[t1:t2]) takes the maximum over last t1 in t2 resolution. For example if you have a query max_over_time((your_query{})[3h:1m]) it tells Prometheus - I want max over last three hours and the points are displayed every 1 minute. You can also do that by hand in Grafana - you can set Last 3 hours, set the Step parameter to 1 minute, and type in your_query{} as the query. Then manually find the max point and here you are - you manually did, what promql.

is equal to

Hope that helps!

Thanks for your suggestions! It is not quite clear to me, however… if I use max_over_time and “Instant” mode, is it guaranteed that I will get the maximum value the metric had per container throughout the entire range? sounds like I will just have MAX value over the last specified interval in the range (unless I can somehow provide that [3h:1m] expression to be always equal to the whole range of the panel)

PS: it looks like “range” is whatever user gets to choose on the dashboard (i.e. datetime “from” / "to); while “interval” is something used by PromQL functions… sounds super weird, I wish they could find less ambiguous wording for non-English speakers :frowning:

To get the max over the time range from the time picker, you can use $__range variable (the syntax will be [$__range:1m] or [$__range:$__interval] (you can also use rate_interval instead of interval))

I have to admit, this actually works. But why on Earth would it not work with Expressions? … sounds like same semantics to me and also should be simpler/less demanding to calculate…

thank you @dawiddebowski !

Tbh I don’t know if it doesn’t work - for me it was much easier done with PromQL / transformations :smile:.