Filtering Alert List label over variable does not work

Grafana Dashboards
1

  • What Grafana version and what operating system are you using?
    Grafana 8.4.6

  • What are you trying to achieve?
    I want to create Dashboards that display different pannels with the live alerts, depending of Severity and Namespace. I want fixed pannel for Severity (one pannel per severity), and dynamic pannel for Namespace.

  • How are you trying to achieve it?
    For Warning alerts, I create a Alert list Pannel with filter Alert instance label equals to variable “$namespace” and “warning”

  • What happened?

No filtering is done (aka all alerts are visible no matter what are their namespace) when choosing a specific namespace out of my variable. However, if I hardcode the filter to namespace=“mynamespace”, then filtering work, and only alerts taken with namespace “mynamespace” are visible.

Note that if I use the same variablefor other pannels in the same Dashboard, there is no problem.

  • What did you expect to happen?

I expect the filtering on label in a Alert Lists to work on variable.

  • Can you copy/paste the configuration(s) that you are having problems with?

Templating:

"templating": {
    "list": [
      {
        "current": {
          "selected": false,
          "text": "All",
          "value": "$__all"
        },
        "definition": "label_values(ALERTS, severity)",
        "hide": 0,
        "includeAll": true,
        "label": "severity",
        "multi": false,
        "name": "severity",
        "options": [],
        "query": {
          "query": "label_values(ALERTS, severity)",
          "refId": "StandardVariableQuery"
        },
        "refresh": 1,
        "regex": "",
        "skipUrlSync": false,
        "sort": 0,
        "type": "query"
      },
      {
        "current": {
          "selected": false,
          "text": "All",
          "value": "$__all"
        },
        "definition": "label_values(ALERTS, equipe)",
        "hide": 0,
        "includeAll": true,
        "multi": false,
        "name": "equipe",
        "options": [],
        "query": {
          "query": "label_values(ALERTS, equipe)",
          "refId": "StandardVariableQuery"
        },
        "refresh": 1,
        "regex": "",
        "skipUrlSync": false,
        "sort": 0,
        "type": "query"
      },
      {
        "current": {
          "selected": true,
          "text": "acceptation",
          "value": "acceptation"
        },
        "definition": "label_values(ALERTS, namespace)",
        "hide": 0,
        "includeAll": true,
        "multi": false,
        "name": "namespace",
        "options": [],
        "query": {
          "query": "label_values(ALERTS, namespace)",
          "refId": "StandardVariableQuery"
        },
        "refresh": 1,
        "regex": "",
        "skipUrlSync": false,
        "sort": 0,
        "type": "query"
      }
    ]
  },

Alert pannel not working as expected:
Important line is

    "alertInstanceLabelFilter": "severity=\"warning\",namespace=\"$namespace\"",


      "gridPos": {
        "h": 14,
        "w": 12,
        "x": 12,
        "y": 10
      },
      "id": 4,
      "options": {
        "alertInstanceLabelFilter": "severity=\"warning\",namespace=\"$namespace\"",
        "alertName": "",
        "dashboardAlerts": false,
        "groupBy": [],
        "groupMode": "default",
        "maxItems": 20,
        "sortOrder": 1,
        "stateFilter": {
          "error": true,
          "firing": true,
          "inactive": false,
          "noData": false,
          "normal": false,
          "pending": true
        }
      },
      "title": "Alertes Warning",
      "type": "alertlist"
    }

Alert pannel working as expected (difference is this line

    "alertInstanceLabelFilter": "severity=\"warning\",namespace=\"mynamespace\"",


      "gridPos": {
        "h": 14,
        "w": 12,
        "x": 12,
        "y": 10
      },
      "id": 4,
      "options": {
        "alertInstanceLabelFilter": "severity=\"warning\",namespace=\"mynamespace\"",
        "alertName": "",
        "dashboardAlerts": false,
        "groupBy": [],
        "groupMode": "default",
        "maxItems": 20,
        "sortOrder": 1,
        "stateFilter": {
          "error": true,
          "firing": true,
          "inactive": false,
          "noData": false,
          "normal": false,
          "pending": true
        }
      },
      "title": "Alertes Warning",
      "type": "alertlist"
    },

Pannel with Namespace variable working correctly:

"gridPos": {
        "h": 8,
        "w": 11,
        "x": 0,
        "y": 1
      },
      "id": 14,
      "options": {
        "legend": {
          "calcs": [],
          "displayMode": "list",
          "placement": "bottom"
        },
        "tooltip": {
          "mode": "single",
          "sort": "none"
        }
      },
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "prometheus"
          },
          "exemplar": true,
          "expr": "SUM(ALERTS{severity=~\"$severity\", namespace=~\"$namespace\"})",
          "interval": "",
          "legendFormat": "",
          "refId": "A"
        },
        {
          "datasource": {
            "type": "prometheus",
            "uid": "prometheus"
          },
          "exemplar": true,
          "expr": "",
          "hide": false,
          "interval": "",
          "legendFormat": "",
          "refId": "B"
        }
      ],
      "title": "Total alertes",
      "type": "timeseries"
  • Did you receive any errors in the Grafana UI or in related logs? If so, please tell us exactly what they were.

No error is displayed

  • Did you follow any online instructions? If so, what is the URL?

No