Links to dashboards with variables and values based on regex

[calestyo]

• What Grafana version and what operating system are you using?
  10.2.3, Debian

• What are you trying to achieve?
  I have a dashboard where a panel (for example network IO) is repeated for 1-n hosts that are selected via a variable (with the data coming from Prometheus).

In the drop down for the variable that selects the nodes, I can either select 1-n single nodes or the special element All.

Problem is now, there are many nodes, an some belong logically together, so ideally I’d want to have a number presets, where each selects a certain group of hosts.

Now, AFAIU, directly creating preset as a special element like All is not possible, i.e. I cannot define a webservers name, which in fact select something like www.example.org and www2.example.org or so.

• How are you trying to achieve it?

I can however make something like a preset by using URLs to the dashboard and specifying 1-n variable values for it, e.g.

http://grafana.example.org/…mydashboard?var-node=www.example.org&var-node=www2.example.org

I can of course make some own webpage, which somehow creates the links and updates them as new nodes are added to groups, but is there some way to get this in grafana itself?

Ideally I’d want something like a home panel, that contains links to other dashobards, and where I can also set via e.g. some reg-ex or perhps some external file, which vars are to be set to which values.

I’ve seen Manage dashboard links | Grafana documentation but that seems to be only very limited and not really do the above.

Thanks,
Chris.

[yosiasz]

@calestyo

belong logically together,

Can you please define this logical grouping? Is there data associated with it or is it just tribal in house knowledge

[calestyo]

It’s just tribal house knowledge, i.e. no information about the grouping in Prometheus or so.

What is in Prometheus are all hostnames, and for some (but not all) groups, the members of a group would have hostnames following a certain pattern (like worker[0-9]+.example.org or so), which is why a solution where I could use (statically configurable) regular expressions against the list of host names (as from Prometheus) would be nice.

[yosiasz]

So the nomenclature used for the host names has a specific pattern that easily lends to be extracted by regex?

[calestyo]

In many cases it is, yes, and for those where it’s not I could use simply a regex that also matches …|^full.quallified.hostname.example.org$.

[yosiasz]

I would think maybe a non prom data source that has this grouping built through cron automation script or manually?

Thinking out loud, a table with parent grouping then hosts as children

When one selects group_a all hosts under that group are used as filter?

But you show a drop down for hosts for visual purpose but under the hood are the hosts passed for filtering

[calestyo]

Uhm, forgive me my ignorance (I’m fairly new to Grafana)… but how would I actually get such data into Grafana (assuming I already had it in some DB)?

[yosiasz]

drop table hostgroups

create table hostgroups(groupname varchar(50), hostname varchar(50))

insert into hostgroups
select 'group_a', 'www.example.org' union
select 'group_a', 'ww2.example.org' 

select
   distinct  groupname as __text,
    stuff((
        select ',' + u.hostname
        from hostgroups u
        where u.hostname = hostname
        order by u.hostname
        for xml path('')
    ),1,1,'') as __value
from hostgroups
group by hostname,groupname

Which gives us the following and we use that in the dashboard as a variable called groups

Then it shows up in our dashboard as follows

So the drop down shows us the group name but notice the value I put in the panel title

This does not have to be done in a database, you could also do it in a csv file but depends how often you have new hosts etc. and maintenance of this csv file manually seems a bit bootleg

[calestyo]

Hey.

First, thanks again.

But I’m still unsure how this (fully) fixes my problem.

So first, I leave out your SQL, cause that is anyway clear how to do (though I’d probably use a simpler csv).

For starters I did that now, i.e.:

• Create a CSV with some columns and rows
• Used the CSV plugin to get that into grafana
• Then, AFAIU, your idea is to use a query variable, right?

So I would have e.g. a CSV like:

group_regex
"storage nodes;^storage-.*\.example\.org$"
"compute nodes;^compute-.*\.example\.org$"

or so, i.e. one column, where __text and __value are separated by some character (here ;) and in the CSV query variable I use group_regex as field and /^(?<text>.*);(?<value>.*)$/ as regex.

That gives me a drop down for the variable, and using ${var:raw} I get my regex (i.e. the one in the CSV, not the one for parsing the CSV).

So I could use ${var:raw}, where previously I used e.g. $node which is a multi-value variable that contains all the host names.
If the value of ${var:raw} is a proper PromQL variable, it should again just select the right nodes.

So far, was that your idea?

Then that would be more like a “real” solution, instead what I had in mind… i.e. just getting some URLs with already the right ?var-name= ... set.

There is however a downside, or at least some things which I don’t know yet how to solve:

• With the above, var and it’s drop down would be the selector for the hosts in my queries. But now it contains only the groups. Is there a way to get both into the same variable (and drop down), my groups and the plain host names (without having to add the separate hostnames into the CSV)?
• Would it be possible - I guess it would - to then select single hostnames and groups at the same time? I.e. by having var be a multi-value variable? AFAIU,grafana would then just concatenate (separated by |) my group regex, and the hostnames (which actually are also regexes, just without meta-chars[0]).
• For my “(dynamically generated) static file with just URLs to the dashboards with the right variabes set via the URL”-idea… I had thought about going further than just doing groups for hostnames.
  For example, we have storage nodes (which would be a group), and on some dashboards, we plot e.g. the disk IO. For those we have a variable the allows to select the devices, for which IO is to be plotted.
  Now it would be nice, if for those I had groups, too, e.g. if one has selected storage-nodes as the host group, I’d perhaps want to see a group system-disks and a group data-disks for the devices variable.
  Right now I’m a bit unsure how one could bring that multiple layers into the current schema.

Cheers,
Chris.

[0] Actually I just realise, that what e.g. Node Full Exporter does to get the host names (which is a query var with Query Type “Label Values” on the label instance on node_uname_info) is probably not a literal string, because the . in the hostnames won’t be taken as literal . but as meta-character .?
If so, is there any way to escape the string?

[yosiasz]

@calestyo

So far, was that your idea?

Yep that was it.

Irregardless of the data source sql, csv same principle

[calestyo]

I played a bit more with that, and it seem it cannot be used as desired.

If the variable holds the regex, then many things break:

• repetition (I don’t get anymore one panel per (matching/selected) host, but all matching time series are plotted in the same panel
• I cannot anymore do e.g. “$var” in the title of a repeated panel, as this is now the regex, and not any longer the concrete host name

[yosiasz]

What are you using the repeater name

[calestyo]

Well for the repetition-variable (i.e. the one in “Repeat options”) it only offers me the plain node, and I cannot override it with something else like e.g. ${node:raw} or so.

For the title value of the panel I tried both ${node:raw} (in which case it prints the plain regex) and ${node} (in which case it prints the “label” of that value (i.e. my “group name”).

And it draws all resulting time series in one panel, whereas I want one panel for each.

[yosiasz]

@calestyo

group_regex
"storage nodes;^storage-.*\.example\.org$"
"compute nodes;^compute-.*\.example\.org$"

for the above data for testing purposes see what it looks like in a dashboard instead of variables. btw which plugin are you using for this or is it just straight manual

Just to vet out that variable

[calestyo]

I’m using that CSV plugin, installed from within grafana itself, as it’s not packaged for Debian.

My csv file is then on some test webserver and contains:

group
storage nodes;^lcg-lrz-dc.*.grid.lrz.de$

with group being merely a header.

The dashboard’s variable is configured like this:

The panel’s queries are:

And if I set the panel’s title to: ${node:raw} I get:

Which I think is just what you get, isn’t it?

[calestyo]

Which plug in are you using?

Cause you seemed to have something different, and you don’t seem to need a regexp for the dashboard variable to properly set label/value?

But even then, it seems your dashboard would give only one panel per regexp, not one per matching host.

What I basically want, when I select a group storage-nodes and that would be a regexp that matches:

lcg-lrz-dc20.grid.lrz.de
lcg-lrz-dc21.grid.lrz.de
lcg-lrz-dc22.grid.lrz.de
lcg-lrz-dc23.grid.lrz.de

The dashboard should result in:

i.e. one panel per matching node, not per selected regexp-group.

[yosiasz]

doesn’t matter which plugin, that is not the issue. let me be more explicit.

use the variable’s value in the dashboard title to vet things out. so in the title of the dashboard add ${node:raw}

What do you see when you do that? let’s stay laser focused as you seem to be going into rabbit holes.

[calestyo]

That’s what I wrote just before.

Last screenshot, there the title is set to ${node:raw} and in the panel title of the screenshot I get the raw regular expression ^lcg-lrz-dc.*.grid.lrz.de$, as you do in your examples.

[yosiasz]

and I am blind! sorry

Ok now let’s add the regex filter manually and see what happens? Maybe you already did that and I totally missed it

[calestyo]

No worries

By “adding manually” you mean using ^lcg-lrz-dc.*.grid.lrz.de$ directly in the query, not via the var?

If so, it gives the same result as with the ${node:raw} in the query, all graphs in one panel. No repetition of panels.