Extracting variables from Prometheus labels, sorting by value

Grafana Configuration
1

I’ve got a couple of related questions regarding variables and sorting.

I’d like to display a fairly large set of single-stat panels, using repeat. These are essentially red/green panels based on the value of a Prometheus query (green = 0, red = > 0).

To improve the usability of this, I’d like to move all the red panels to the top of the dashboard on refresh.

There currently seems to be no mechanism for ordering, so I’ve attempted two use two separate solutions to this.

  1. Use 3 template variables. One is getting all labels for the query, then two separate queries for success/failure and two separate repeating panels.
    e.g
    $checks = label_values(health_check_status, system)
    $failing checks = query_result(sum(health_check_status{system=~"$checks", observe="yes"}) by (system) > 0) with regex {.*?system="(.*)".*
    $passing_checks = query_result(sum(health_check_status{system=~"$checks", observe="yes"}) by (system) == 0) with regex {.*?system="(.*)".*

This works great on dashboard load, however there are a couple of issues:

  • variables aren’t refreshed on the dashboard’s refresh interval, so nothing is ever re-ordered. This seems to required a dashboard refresh. Is there any way of triggering these to be refreshed automatically?
  • if filtering by a variables, there’s an empty singlestat panel with no value if all of the series are 0, or > 0.
  1. Try to use query_result(sum(health_check_status{observe="yes"}) by (system)) and sort numeric (descending). This doesn’t work however as I need to extract the labels using regex, not the values. Is it possible to sort based on the values of the query, but extract the text labels?