Repeated panel with multiple label_values

Grafana
1

I have the following set of metrics being returned from Prometheus for PostgreSQL databases

up{cluster="alpha", exp_type="pg", instance="192.168.122.16:9187", job="ip16_pg1_alpha"} 1
up{cluster="alpha", exp_type="pg", instance="192.168.122.26:9187", job="ip26_pg2_alpha"} 1
up{cluster="alpha", exp_type="pg", instance="192.168.122.36:9187", job="ip36_pg3_alpha"} 1
up{cluster="bravo", exp_type="pg", instance="192.168.122.16:29187", job="ip16_pg1_bravo"} 1
up{cluster="bravo", exp_type="pg", instance="192.168.122.26:29187", job="ip26_pg2_bravo"} 1
up{cluster="bravo", exp_type="pg", instance="192.168.122.36:29187", job="ip36_pg3_bravo"} 1

We have an overview panel that repeats a stats panel based on the label_values(“job”) value so we have one panel per database cluster.

pgoverview
pgoverview1617×325 49.2 KB

I added a new “cluster” label to each metric to better tie the systems in a replication cluster together. Note that the fact that the cluster is in the title currently is just because I automatically name the nodes to include the cluster name to ensure uniqueness by default (it’s in the “job” value). Our users may not do that, so I cannot rely on the cluster name actually being in the node/job name.

I’d like to group them together in the output by cluster for this overview dashboard, But I don’t see a way to repeat based on multiple label values. Or to at least be able to use other label values other than the one I’m repeating on. Is there a way to extract the “cluster” label’s value for each metric out and add that into the title? Not sure if that would solve my issue though, since I’m assuming it just puts the panels in order based on the actual repeated value.

So basically I’d like the titles of the panels to be “cluster - job” so the resulting panel titles would be as follows, and then also sort them in this order:

alpha - ip16_pg1_alpha
alpha - ip26_pg1_alpha
alpha - ip36_pg1_alpha
bravo - ip16_pg1_bravo
bravo - ip26_pg1_bravo
bravo - ip36_pg1_bravo

Thank you!

2

Hello Keith, not quite sure if I got your question right. Maybe you could draw something to help imaginate. thank you

3

Basically I’m trying to make the title of a repeated Stat panel use the “cluster” and “job” labels to create the title value. See initial post for examples of my metrics and the exact title values I’m trying to achieve.

4

hello i mean a graphic could help

5

This topic was automatically closed after 365 days. New replies are no longer allowed.