How do dashboard variables affect SLI alerts

Grafana Dashboards
1

I have a bunch of existing SLOs with dashboards and alerts. Currently they are essentially Success metric=all 2xx responses Total metric = all responses. We are deciding if we should be excluding 4xx responses from the Total metric so would like to have a dropdown on the dashboard to switch whether they are included.

Following advice from Grot AI and Chat GPT - I added a Custom variable to the dashboard which had two differet regex values and then added it to the Total Metric (http_server_duration_milliseconds_count{job="specific/component",http_status_code=~"$include_4xx"} where include_4xx is the literal regex. It’s crude but works for the dashboard

My question is, presumably include_4xx won’t be set outside the context of the dashboard, such as when calculating whether to send an SLO alert. What will happen here? Will it just fail? Or have I misunderstood how the dashboards/alerts work?

2

Actually I can see that adding the variable means “run queries” at the bottom of the Edit SLO page no longer works, so presumably this approach breaks the SLO outside of the dashboard

3

Hi,

Variables are closely tied to the dashboard. So yes, if you run such a query:
http_server_duration_milliseconds_count{job="specific/component",http_status_code=~"$include_4xx"}
in e.g. Explore mode it will not as much fail, as return empty resultset as $include_4xx will be empty string and I assume all time series of this metric have status code. As for alerts - there’s also no way of including variables in them. Alerts require a constant query but you can always do two alerts or even one alert with two queries and just switch which query is the alert condition.