DashboardQueryRunner failed - failed to fetch

Grafana Dashboards
, ,
1

  • What Grafana version and what operating system are you using? 9.1.0

  • What are you trying to achieve? A healthy dashboard without any errors

  • What happened?

I have setup a custom dashboard for monitoring the application status using blackbox exporter and node exporters with prometheus as datasource and it was working fine. Recently I observed that a notification pop out stating “DashboardQueryRunner failed - failed to fetch” and “unexpected error”. The panels stopped working (Shown NA for data).Then after 2 min, when I do a manual refresh of the dashboard, the error got cleared and got to normal. Don’t know the exact reason behind.

Also to note that the prometheus gives a clear report on the query during the said period without any flaws. One more observation is that the other dashboard in the same grafana was working fine during the period of observation.

My applications are running as docker containers.

Do anyone faced the issue and what was the resolution.

  • Did you receive any errors in the Grafana UI or in related logs? If so, please tell us exactly what they were.
"DashboardQueryRunner failed - failed to fetch"
"unexpected error"

grafana error
grafana error1280×1143 112 KB

1 Like
2

Hi @vinod9devops,

Thanks for opening this post.

Could you please change the default log level to debug mode?

To do this go to the server where grafana is installed and edit the configuration file (/etc/grafana/grafana.ini)

There you will this section;

#################################### Logging ##########################
[log]
# Either "console", "file", "syslog". Default is console and  file
# Use space to separate multiple modes, e.g. "console file"
;mode = console file

# Either "debug", "info", "warn", "error", "critical", default is "info"
;level = info

Set the parameter level to debug and remove the colon in the beginning i.e.

level = debug

Then restart the grafana service and resent the logs once the issue occurs again.

3

@usman.ahmad Thanks for that info. Updated to debug mode. But I can’t simulate this error now. It happens in a while. I faced the error once a month back and now. How to dig the root cause. Any leads … will be appreciated.

4

Hi @vinod9devops

Thanks for providing more details.

I would suggest to keep the log level to debug mode and wait till this happens and update this post (if you are good on the disk space).

Also, I see that you are using Grafana version 9.1.0 and would recommend to use the current stable version 9.5.3 as its more bug fixes than your current version. Maybe it also solves this Dashboard error as well !!

5

We are also facing the same error. We ran the grafana 9.4.7 in debug mode but there are no errors , debug logs relating to why this is failing.

6

Hi @here,

Is this similar to your use case:

Are you also using single-text variable input OR something else?

7

Hi,
I face a similar issue after the upgrade from Grafana 8.4.11 to 9.5.13.
The error is displayed when I switch between organizations, and the dashboards that I used during the switch, contain only dashboard list panels and no variables at all.
I cannot say that it is reproduced always, but it may be annoying for users.