Variables are not populated unless manually updated

[berwangerkris]

Hi,

I am running Grafana v9.1.0-73288pre (046a2602ff) in a Docker Container. For a while now I have set up a Dashboard to visualize and analyse my rock climbing training.

My Docker Container updates automatically using Watchtower. Since more recent updates (unfortunately, I cannot determine exactly which update) the automatic variable population when the dashboard is loaded does not work for me any more. In the past, I loaded the dashboard and the queries associated with the used variables would execute in the background and populate drop down menus, so variable values could be chosen to look into the climbing data. For example, I was able to choose between “all”, “climbing” or “bouldering” data when the dashboard was loaded. But when I load the dashboard now, the variable associated with the same drop-down menu has no values. This can be seen in the next screenshot.

When I have a look at the variables under dashboard settings, I can see that there are no values associated. When I manually hit the “update” button, the underlying query seems to be executed and the variable populated.

Afterwards, the associated drop-down menu is populated and the dashboard can be used as normal/before.

To populate all drop-down menus, I have to repeat the manual update for each associated variable. Then the whole dashboard is useable again.

My preliminary research did not come up with any solutions, or anyone else reported this or a similar problem. I have 2 thoughts on this (but please keep in mind that I really am a noob in this):

1. One of the last Grafana updates “broke” something, which results in the variable queries not running automatically when a dashboard is loaded.

2. There was maybe a setting introduced that the automatic execution of the variable queries has to be actived? Which I could not find, but maybe I am blind.

Any help or suggestions are highly appreciated, Thanks!

[melori.arellano]

@berwangerkris this isn’t expected, I have a couple of ideas you could try to repair the dashboard. The variables are stored in the database, so it’s possible something happened during a recent update.

• What happens if you export the dashboard json and then import the dashboard? Use the toggle export for sharing externally, which will prompt you to select an input datasource when you import the dashboard - I suggest creating a new dashboard ID and name when you import so you can verify how it behaves.

[berwangerkris]

@melori.arellano Thank you for the suggestion!

I tried your export and import with new ID and name suggestion. I see the same behavior on the newly imported dashboard. When I load the dashboard, all variables are unpopulated. Only after I manually update the variables in settings, the variables (drop-down menus) are populated.

What is also weird: I have a variable var_grading, which is of type custom and lets me choose the grading system that the data in the dashboard should be converted to (climbing routes have different grading systems). According to the value of var_grading the variables var_from and var_to are populated. Both of these variables have exactely the same underlying query:

SELECT (CASE
WHEN ‘$var_grading’ = ‘french’ THEN grade_fr
WHEN ‘$var_grading’ = ‘uiaa’ THEN grade_uiaa
WHEN ‘$var_grading’ = ‘yds’ THEN grade_yds
WHEN ‘$var_grading’ = ‘font-scale’ THEN grade_fontscale
WHEN ‘$var_grading’ = ‘v-scale’ THEN grade_vscale
END)
FROM template_sorter

However, only var_to will be updated according to a change in var_grading. var_from only changes when I manually update through settings.

Is there a log I could look into to debug?

[melori.arellano]

@berwangerkris I like to look at developer tools in the browser (the network tab) to see the requests and responses sent by the page (debug not required for this).

If you want to enable debug logging edit the configuration file grafana.ini:

[log]
# Either "console", "file", "syslog". Default is console and file
# Use space to separate multiple modes, e.g. "console file"
mode = console file

# Either "debug", "info", "warn", "error", "critical", default is "info"
level = debug

Then restart grafana for the setting to go into effect.

[berwangerkris]

@melori.arellano I looked into the network tab of the dev tools in chrome. When loading the page, I can see some queries that are stated as status cancelled and there are errors present. These errors concern a Web Socket connection:

(I disguised my privat URL in the screenshot above)

I can export an HAR file. Would that help, or gives the screenshotted error enough information?

---

Edit: I saw a similar error description in this thread [Link]. Posters hinted that the correct root-URL has to be set under the [server] section in the Grafana config. I already had this URL set to my full domain (https://XYZ.ZYX.de/) for redirects to work properly. Also, I am running this Grafana Container on a Synology Diskstation and using the Synology built in Reverse Proxy. I thought I add this information for clarification, maybe.

[berwangerkris]

What I also see in the network tab of the dev tools is the following query failures:

These failed queries appear when the website is loaded.

Also, when I select a value for a variable X in a drop-down menu, that should trigger a query for variable Y to repopulate the variable Y according to the value selected for variable X, I can see a failed query in the network tab dev tools:

When I navigate into the entry in dev tools under response, it says “Failed to load response data: No data found for resource with given identifier”. Which I find weird, because I can manually trigger those queries fine (as describe above), so the data is definitely there.

[DavidMonroe]

Same issue, variables are not populated until queries are run manually from the variable editor page. Chrome dev tools shows error message:

But with no additional info, and nothing in /var/log/grafana/grafana.log

Think I “fixed” it by adding a hidden dummy Option variable which I reference in the query at the top of my variable dependency hierarchy, which seems to trigger the queries properly, tricking the sql queries into thinking the need to wait for the UI to load, I guess, and not running too early.
Also note that when refreshing a page that’s working, it’ll continue working due to the default values baked into the url. Only starting from a fresh page with no values selected do you get the error.

[freesinger]

Same issue as @DavidMonroe and @berwangerkris encountered, curious how @DavidMonroe solved this issue by tricking UI?

[DavidMonroe]

I’m back here again because my hack isn’t working anymore after copy the dashboard and changing a number of things, not sure why. Maybe it never worked.

@freesinger incase you’re desperate, you just need to add a custom variable and reference it in your other variable queries (in an inconsequential way like checking if $hackVar = 0 or $hackVar != 0). This will reorganize your variable hierarchy, which determines which queries need to rerun when, which might have fixed my problem a month ago but no longer.

But anyways @grafana, problem still here. Put simply, variables referencing other variables don’t cascade and rerun when they’re supposed to. The “Hierarchy” appears correct when I view it, and there’s no errors in chrome console. Currently my staff can’t use our dashboard, and I’m reorganizing our database so we don’t have to rely on grafana dashboard.

Appreciate all the volunteers who work on this stuff btw, otherwise the product is great.

[DavidMonroe]

Bumping again, bug still around, plus I think I figured out what’s wrong and a hack-fix for it

Basically, if a query variable is used in multiple other query variables, it only updates the first one. So if I have a picker “PersonID” and variables “PersonsFriends” and “PersonsEnemies”, when you pick a PersonID, PersonsEnemies should rerun, but it won’t.

Fix: Ensure a linear chain of all your variables, eg by including “-- $PersonFriends” and the end of your PersonEnemies query. Note that dependencies still count commented out variables, so you don’t actually have to modify your queries, just tack a comment on. Additionally, I have all variables set to update on time range change, and a dummy variable ‘select now()’ at the top of my hierarchy, with that dummy variable moved to the bottom of this list and hidden (these things may or may not matter).

[stephenjanes]

I am running into this issue as well. I am using AWS Managed Grafana v9.4.7. In my instance my source is AWS Timestream. I have a variable called “Tags” which users of the dashboard use to select tags they wish to trend. My trend panel uses the following WHERE Clause:

Where $__timeFilter and fulldesc.F in ($Tags )

One initial load of the dashboard the Trend Panel throws this error

Line 15 is the line of the WHERE clause and the query inspector shows the following:

If I use the dropdown for the Tags variable, unselect the tag and reselect it, the trend then works and the query inspector shows the following (The trend also works if I select multiple tags):

Notice that now it has single quotes around it.

I need a way for the $Tags variable to always have the single quote around it whether it is the initial load, a single tag is selected, or multiple tags are selected. Please help.