Custom Variables: generally failing for my use-case

[notjames]

• What Grafana version and what operating system are you using?
  Currently using 11.4.0

• What are you trying to achieve?
  We use Grafana in EKS. We use RDS instances. We use cloudwatch as a datasource in grafana to pull metrics from RDS. We are using a centralized grafana to pull metrics from multiple clusters. Our clusters run in different sub-accounts and our RDS instances run local to those sub-accounts.

In the dashboard, we have a drop-down to allow operators to select the cluster with the metrics they want to view. So I need to map the cluster (victoriametrics) to the cloudwatch RDS instance name and then write a “query” that will chain the variables so that the dashboard pulls metrics from the correct Cloudwatch RDS instance correlated with the cluster.

• How are you trying to achieve it?

I’ve created four variables so far:

1. a standard variable for cluster pulled from prometheus (actually victoriametrics without the VM plugin for now)
2. an instance_map (db instance map)
   
   
3. a join variable which maps the cluster to the instance_map (this is the one that is failing miserably). The screenshot below shows empty fields and this is because grafana nullifies or empties the fields as soon as I save the dashboard or run the query (and typically with no errors). I’ll show a screenshot with how I’m trying to do this and then one with what happens after grafana empties the fields:
   
   
   
   
   
4. Final variable for cloudwatch to use the selected instance
   
   

• What happened?
  Grafana clears the fields in instance_map and occasionally yields an error which I was unable to capture. The error is inconsistently shown. I’m not able to use the mapped db instance to cluster relationship.

• What did you expect to happen?
  I expected the variable interpolation to work so I can relate the db instances to cluster names so that I can view cloudwatch telemetry for these DB instances.

• Can you copy/paste the configuration(s) that you are having problems with?
  If they are required, I can try and get them and sanitize what needs to be sanitized if necessary.

• Did you receive any errors in the Grafana UI or in related logs? If so, please tell us exactly what they were.
  I did, but I was not able to capture them and they show up inconsistently such that I haven’t seen one in a while despite the failures described. If memory serves, it was something about the data source and the labels_value function not being available or valid or something, which made some sense to me. Where I’m not grasping thing is that I’m trying to use a variable as a source for a label which will then be used as a variable for a data source from cloudwatch. If I can reproduce the error, I’ll make sure and get it in this post.

• Did you follow any online instructions? If so, what is the URL?
  Add variables | Grafana documentation

Variables | Grafana documentation

https://community.grafana.com/t/access-to-the-value-of-a-custom-variable-key-value-from-another-
custom-variable-value/118039/8

Select value from custom key/value variable in Grafana

[jangaraj]

@notjames

So I need to map the cluster (victoriametrics) to the cloudwatch RDS instance name and then write a “query” that will chain the variables so that the dashboard pulls metrics from the correct Cloudwatch RDS instance correlated with the cluster.

Why? IMHO you have everything in that “map”:

prometheus_cluster_name : cloudwatch_rds_instance_name

[notjames]

@notjames

I’m not sure what I said that makes you think that correlation already exists. There has to be a map of the database to the cluster. That doesn’t currently exist. What made you believe that I said that and I’ll correct it or I just don’t understand what you’re getting at.

I’m not sure what I said that makes you think that correlation already exists. There has to be a map of the database to the cluster. That doesn’t currently exist. What made you believe that I said that and I’ll correct it or I just don’t understand what you’re getting at.

VM has cluster name – “cluster1”
RDS has DB instance name – “db1”

custom variable has a map essentially of: {cluster1: db1, cluster2: db2}

then I need to find a way to “glue” that into a query so that cloudwatch datasource can access the db instance “db1” based on the cluster selected in the variable (chaining)

[jangaraj]

Why “glue”? that “map” is key/value pair and you can access both independently.

Accessing variable value when using key/value variable

:text is for key (nice human name - Rules Triggered), you need :value for value (human unfriendly, machine names/ids - rules_triggered).

[notjames]

I apologize. I’m so confused at what you’re getting at. Would you please explain?

There is nothing else from which I can pull that does that.

prometheus_cluster_name: cloudwatch_rds_instance_name

doesn’t make sense to me because it’s not true. In my scenario:

cloudwatch_rds_instance_name == db1

and

prometheus_cluster_name == cluster1

I currently ONLY have the map that relates these two things together and cloudwatch needs the name of the DB instance.

so victoriametrics has cluster_name
the map has the db instance tied to the cluster_name

(pseudocode)

cloudwatch_instance = instance_map['cluster_name']

or in terms of grafana: `

cloudwatch_instance = $instance_map:text[$cluster]`

but I’m not sure how to accomplish this and everything I’ve tried so far, fails.

My problem is that I am not able to get the variable map to work, like at all. So I’m confused why you keep telling me the same thing, which doesn’t make sense and it doesn’t answer my question, and it doesn’t help me get the result I’m seeking. Please help me understand.

:text is for key (nice human name - Rules Triggered), you need :value for value (human unfriendly, machine names/ids - rules_triggered).

I’m aware of this. The question is not related to how to dereference the map. The problem is how to get a solution to what I’m trying to accomplish, and I currently believe the way I’m trying to do this is at least one way if I could just get the ability to properly reference the variable

I’ve showed you the screenshots and I believe I’ve provided enough information to at least give someone the ability to reproduce something in order to provide some guidance.

perhaps:

@jangaraj

IMHO you have everything in that “map”:

…perhaps what you’re getting at here is that the map has what I need. And YES. I know it does. Everything I’ve tried in order to USE the map does not work. Grafana keeps nullifying the fields suggesting that I’m doing something wrong.

[jangaraj]

Create a custom variable, name it keyvalue pls, so you don’t stick to “map” structure in your mind.

Then create a text panel (just for your visualization/debugging) and watch how text in text panel is changed based on the selected dashboard variable:

So then use ${keyvalue:text} in Prometheus queries (if you need that) and ${keyvalue:value} in the CloudWatch queries (if you need that).
What is not working in this setup for your case? What kind of “glue”, “variable chaining” is needed in this setup for your case?

[notjames]

Ah! Very nice. This is better and more understandable instruction for me to see your angle of attack. Thank you.

Unfortunately, I don’t think I can use this because you’re using the key of the keyvalue pair variable as your cluster name. In this use-case that would be a statically generated cluster name, which is not scalable. I’m trying to use cluster from dynamically generated variable from a victoriametrics query on the up metric extracting the cluster label and then map that to the keyvalue variable on the cluster name. This is the chaining that is failing.

Let me know if that makes more sense and if I’m still just not getting it. I appreciate your help.

[jangaraj]

@notjames

In this use-case that would be a statically generated cluster name, which is not scalabl

Yes, that’s based on your original variable.

So where and how are stored cluster/db relations, when you want to have that dynamic and not static?

[notjames]

@jangaraj

So where and how are stored cluster/db relations, when you want to have that dynamic and not static?

We’re not sure yet, but this method is the least non-scalable method we can use right now.

You gave me some ideas about things which caused me to realize that I was missing a key component. That caused me to have to approach this in a slightly different way. The problem remains the same. I still need to know how to chain custom variables only now it’s data source name instead of db instances…

We’re using two separate cloudwatch data sources for two separate AWS sub-accounts, so now instead of using some way to dynamically choose DB instances, I need to dynamically select data sources based on the same cluster selector.

Instead of getting deep in the weeds on that, please answer this. Is it possible to chain variables on a custom variable in grafana dashboards?

Add variables | Grafana documentation suggests that chaining variables is a supported thing, but I’m not getting good results.

[yosiasz]

@notjames

Is it possible to chain variables on a custom variable in grafana dashboards?

No it is not possible to chain Custom variable

[notjames]

Thank you @yosiasz

Is there any good documentation that shows how that would be done for the cloudwatch data source? To reiterate, the following is what I’m attempting to accomplish

We’re using two separate cloudwatch data sources for two separate AWS sub-accounts, so now instead of using some way to dynamically choose DB instances, I need to dynamically select data sources based on the same cluster selector.

Sorta what I put in my earlier posts is what I’m doing now, but again, it’s not working ™.

I’ve created two data sources. For our purposes, let’s call them cw_account_1 and cw_account_2

What I need:

In my dashboard, I’ve created the following:

…which will of course not work. And you can’t reference the $cluster variable in the panel data source like one would do with a normal hashmap dererefence IE $datasource_map[$cluster]

so it seems I need to create a relationship variable:

this is the part that doesn’t seem to work:

I considered approaching this the way @jangaraj suggested, but now that I need to shift methods a little, I don’t think what was suggested will for this use-case.

[yosiasz]

@notjames

Is there any good documentation that shows how that would be done for the cloudwatch data source?

You cannot do that with Custom variables (dynamic change, chained stuff) so there is not documentation

But you can look at the following variable type

then data source option you would select all of the Cloud Watch ones.

otherwise you could automate it all by create a cron job (ETL) that scrapes your different instances into a database such as sqlite or whatever that has the relationships.