Variable to show aggregated timeline only on "All"

  • What Grafana version and what operating system are you using?

Grafana v10.4.15 on Ubuntu Server 22.04

  • What are you trying to achieve?

I want to make variable that would:

  • Show metrics in panel depending on which labels are selected (e.g. show host1 and host3 when they are selected in the variable)
  • Only if “All” is selected, sum all the labels together so it would display one aggregated timeline instead of all of the timelines one on top of another.
  • How are you trying to achieve it?

I successfully set variable for label value, selected metric, selected label value and checked the “Include All option”, but the All option shows all timelines on top of each other. I want to have it aggregated (sum) into one timeline

I also tried to do the sum beforehand in the query. It worked kind of well but when i selected multiple labels to show, it simply shows aggregated timeline.

  • What happened?

Including the sum into query worked better but when i selected multiple labels, it showed one aggregated timeline instead of showing 2 timelines one on top of each other

  • What did you expect to happen?

When i select one label, it should show one timeline. When i select multiple, it should show multiple lines in the same panel. When i select “All”, i want it to aggregate all timelines into one.

  • Can you copy/paste the configuration(s) that you are having problems with?

Sure:

{
  "annotations": {
    "list": [
      {
        "builtIn": 1,
        "datasource": {
          "type": "grafana",
          "uid": "-- Grafana --"
        },
        "enable": true,
        "hide": true,
        "iconColor": "rgba(0, 211, 255, 1)",
        "name": "Annotations & Alerts",
        "type": "dashboard"
      }
    ]
  },
  "description": "REDACTED",
  "editable": true,
  "fiscalYearStartMonth": 0,
  "graphTooltip": 0,
  "id": 120,
  "links": [],
  "panels": [
    {
      "datasource": {
        "type": "prometheus",
        "uid": "raInckNGz"
      },
      "description": "",
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "palette-classic"
          },
          "custom": {
            "axisBorderShow": false,
            "axisCenteredZero": false,
            "axisColorMode": "text",
            "axisLabel": "",
            "axisPlacement": "auto",
            "barAlignment": 0,
            "drawStyle": "line",
            "fillOpacity": 100,
            "gradientMode": "opacity",
            "hideFrom": {
              "legend": false,
              "tooltip": false,
              "viz": false
            },
            "insertNulls": false,
            "lineInterpolation": "smooth",
            "lineWidth": 1,
            "pointSize": 5,
            "scaleDistribution": {
              "type": "linear"
            },
            "showPoints": "auto",
            "spanNulls": false,
            "stacking": {
              "group": "A",
              "mode": "none"
            },
            "thresholdsStyle": {
              "mode": "off"
            }
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              }
            ]
          },
          "unit": "Bps"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 11,
        "w": 24,
        "x": 0,
        "y": 0
      },
      "id": 1,
      "interval": "30s",
      "options": {
        "legend": {
          "calcs": [],
          "displayMode": "list",
          "placement": "bottom",
          "showLegend": true
        },
        "tooltip": {
          "mode": "single",
          "sort": "none"
        }
      },
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "raInckNGz"
          },
          "disableTextWrap": false,
          "editorMode": "builder",
          "expr": "rate(gost_service_transfer_output_bytes_total{host=~\"$Grouping\"}[$__rate_interval])",
          "fullMetaSearch": false,
          "hide": true,
          "includeNullMetadata": true,
          "instant": false,
          "interval": "",
          "legendFormat": "{{host}}",
          "range": true,
          "refId": "A",
          "useBackend": false
        },
        {
          "datasource": {
            "type": "prometheus",
            "uid": "raInckNGz"
          },
          "disableTextWrap": false,
          "editorMode": "builder",
          "expr": "sum(rate(gost_service_transfer_output_bytes_total{host=~\"$Grouping\"}[$__rate_interval]))",
          "fullMetaSearch": false,
          "hide": false,
          "includeNullMetadata": true,
          "instant": false,
          "legendFormat": "Aggregated",
          "range": true,
          "refId": "B",
          "useBackend": false
        }
      ],
      "title": "Download rate",
      "type": "timeseries"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "raInckNGz"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "thresholds"
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              }
            ]
          },
          "unit": "decbytes"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 8,
        "w": 6,
        "x": 0,
        "y": 11
      },
      "id": 2,
      "options": {
        "colorMode": "value",
        "graphMode": "area",
        "justifyMode": "auto",
        "orientation": "auto",
        "reduceOptions": {
          "calcs": [
            "lastNotNull"
          ],
          "fields": "",
          "values": false
        },
        "showPercentChange": false,
        "textMode": "auto",
        "wideLayout": true
      },
      "pluginVersion": "10.4.15",
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "raInckNGz"
          },
          "disableTextWrap": false,
          "editorMode": "builder",
          "expr": "sum(increase(gost_service_transfer_output_bytes_total[24h]))",
          "fullMetaSearch": false,
          "includeNullMetadata": true,
          "instant": false,
          "legendFormat": "__auto",
          "range": true,
          "refId": "A",
          "useBackend": false
        }
      ],
      "title": "Downloaded over last 24 hours",
      "type": "stat"
    }
  ],
  "refresh": "1m",
  "schemaVersion": 39,
  "tags": [],
  "templating": {
    "list": [
      {
        "allValue": "",
        "current": {
          "selected": true,
          "text": [
            "All"
          ],
          "value": [
            "$__all"
          ]
        },
        "datasource": {
          "type": "prometheus",
          "uid": "raInckNGz"
        },
        "definition": "label_values(gost_service_transfer_output_bytes_total,host)",
        "hide": 0,
        "includeAll": true,
        "label": "",
        "multi": true,
        "name": "Grouping",
        "options": [],
        "query": {
          "qryType": 1,
          "query": "label_values(gost_service_transfer_output_bytes_total,host)",
          "refId": "PrometheusVariableQueryEditor-VariableQuery"
        },
        "refresh": 1,
        "regex": "",
        "skipUrlSync": false,
        "sort": 0,
        "type": "query"
      }
    ]
  },
  "time": {
    "from": "now-3h",
    "to": "now"
  },
  "timepicker": {},
  "timezone": "browser",
  "title": "Surfshark",
  "uid": "bediv97gv91q8c",
  "version": 14,
  "weekStart": ""
}
  • Did you receive any errors in the Grafana UI or in related logs? If so, please tell us exactly what they were.

Not really.

  • Did you follow any online instructions? If so, what is the URL?

No.

What is your dataaource

Prometheus. Typical Grafana-Prometheus stack

can you use logical operators to do an if else kind of thing in promql?

also if you are doing aggregation, will it have also be aggregated by time in order for this to be used in time series viz?

You can have two ways:

  1. less cheating but not exactly what you want - you can set stacking in your time series visualization to Normal - you’ll always have the sum on Y axis, but the plot will be composed of multiple time series
  2. cheaty as hell but will (probably) do what you want (and requires infinity plugin). What you can do here is have your original variable set Custom all value for .*:

    then create another variable (I called mine grouping) which will use infinity plugin with configuration like this:

    this variable will be equal by (cluster) if your original variable is set to anything other than .*.
    since Grafana doesn’t really care if the expression is correct, you can append grouping variable to your query like:

sum(rate(host_cpu{host=~“$host”}[5m]))$grouping

If you set host variable to specific hosts it will append by (host) to the end of the query. If you set All (with custom all value set to .*) it won’t append anything ( == it will append an empty string).

Notice that I didn’t have host label, so act accordingly (in my screens and answer change cluster to host).

Worked for me (at least I think so).

1 Like

AFAIK promql doesn’t have anything like if else. Also promql returns time series by default, so you don’t have to aggregate by time :smile:

1 Like

Amazing job! Worked like a charm! It seems grafana is a bit buggy because when i re-named the original variable to “original_variable”, it didn’t work but when i again renamed it into “host” it started working like i wanted.

PS: I had to rename the variable because i already had the original variable named “Grouping” before.

Just a small additional bonus question, is there a way to “append” a new option into original variable? I’m thinking about showing all the timelines at once but not summing them - so i would have option “All” (summed up into one timeline like you already did), “All - timelines” (displaying all the timelines at once but not summing them - so it should use by (host) i guess), and then other options where every option is one host. It is just a extra question - this is already perfect :slight_smile:

I don’t think so - you need the .* as custom all option to already do the non-grouping. You could have another variable that would decide if the grouping for all should be enabled or not - and add it as the second field in json and have the condition in Parsing options include that too. But I don’t think you could do that without another variable.