I couldn’t find the solution for this for a while and decided to finally post in here to ask for some guidance. Any help would be appreciated.
I have a setup as below:
2 Loki data sources (practically identical)
1 Grafana where both DB’s are configured as data sources
— Variable database:
Variable Type: Data source
Data source type: Loki
Multi-value and Include all
— Variable hostname:
Variable type: Query
Query data source: $ database
Multi-Value and Include All
The problem I’m facing is; all panels/queries on the dashboard work fine when I choose one of the database from the dropdown menu. But if I choose All (or both databases), it brings data only from the first database, not bringing anything from the other database.
When I’ve sniffed the traffic on the other database, it seems like Grafana is querying the database with the values which returned from the first database and naturally, gets no answer. So it seems like Grafana is asking the first database first, gets the values from there which is intended but then instead of asking the same query to the secondary DB, it is adding the returned values from the first DB on the query.
One of the queries I use is as below:
Data source: Mixed
Query A: $database
sum by (hostname, level) ( count_over_time({job=“syslog”, hostname=~“$hostname”, level=~“$loglevel”, level!~“informational”}[1m]) )
I’m sorry for putting it into words so goofly but I’m not really a coder at all.
That sounds like a configuration in your variables. That only controls which data source to query when setting up the variable, not where the graphs or dashboard gets data from.
Each query from each graph has its own data source.
Sadly I did not understand what you mean exactly though; Cannot you use a single query in a single graph/panel to fetch data from multiple sources and combine them on a single output?
Or do you mean the variables I’ve set up ($database, $hostname) is only valid for the “dropdown menu” and not for the queries inside a panel?
If that is the case, how can I modify the panels? Or can I at all?
Edit:
Not sure if that was what you meant but I’ve created a new variabled ($merged), and added $hostname1 and $hostname2 under it. Now I do have the merged values under a single variable which was my intention.
Only problem remaining is how to reconfigure the queries to use that $merged:
I need to use a query to “populate” $hostname1 and $hostname2 first in order for $merged to be populated. Because right now, on first dashboard load, $merged returns empty. I am using a query which includes hostnames first so that $merged can be used for the next step.
That “multi-value and include all” doesn’t make sense, because panel queries work with just one datasource. Yes, variable configuration UI will allow you to configure that, because that’s standard configuration option for all variable types. But it doesn’t work like that you select multiple/all datasources and then panel will query all those selected datasources.
Do you mean that on a single query only 1 data source will be used and in order to use multiple data sources on a single panel/graph, you need to add multiple queries?
I have made something as below, 2 different but similar panels. Does it make sense?
One the first panel, I have $database selected as the data source. And they query for $merged which is a new variable consisting of $hostname1 and $hostname2.
When looking at it, it doesn’t make much sense because they are practically the same query, right?
On the second panel, I have Mixed data source. And on 2 queries, I’ve manually given the data sources. It will check the new variable $mergedhost consisting of $host1 and $host2.
Edit:
The problem I’m facing is:
-If I choose Mixed data source and use 2 queries, dedicated to each data source, it will work and I can filter out on the dashboard (the selected “hosts” I mean). But because it is hard coded to query for both databases, $database becomes useless. I want it to be useful as such as other variables because I want to be able to just choose the data source on the dashboard to see only data from the selected data sources.
For instance, someone should be able to just choose database1 so that the graphs will show values (host, hostname etc.) belonging to that database.
User can select from multiple (cloudwatch datasources, yours Loki “database” variable) - but it can select exactly one datasource (not multiple, not all - because that doesn’t make sense), then he can select variables, which are related only to that particular datasource, e.g. region or loadbalancer (yours host or hostname).
I’m able to populate a “merged hostname” variable where all hostnames from all sources are selectable which cover most of the use cases.
But I want to ask this as I couldn’t fully understand your use case on the scenario:
Why would you select your data source at all, cannot you just select your Region? Or is it so that the same Region might have data from multiple data sources?
On my case every value (host) is unique therefore a certain host will be only in a single data source. If that is the case than I can understand your need for Data Source. On my case, I think, I will not be able to make use of it.
And you have generic dashboard - user can select Loki datasource (KKB or EQX, just one, not multiple, not all) and then host variable will be populated with host values, which are available only in that particular datasource. Then panels/graphs will show data only for that one selected particular host.
What is not acceptable on this dashboard concept for your needs?
That indeed will cover the needs for the end user; it touches all the bases. I’ll do this today once the maintenance is over; cannot reach the servers right now.
Thanks a bunch for the detailed explanation. I’m still trying to grasp the concept of it. It is kinda overwhelming to start with this after 40 years
