Grafana Dashboard queries don't work in browser but work through curl/postman

Grafana Dashboards
1
  • What Grafana version and what operating system are you using?
    11.3.0 on OKD (OpenShift upstream)
  • What are you trying to achieve?
    getting dashboard to show graph values for specified time range
  • How are you trying to achieve it?
    TIG-Stack deployed on Openshift. Dashboard connected to influxDB data-source.
  • What happened?

Setup

I initially had TIG-stack (Telegraf, InfluxDB and Grafana) deployed in a linux server where it worked as expected. I’ve recently deployed it on OKD (OpenShift upstream). I have several dashboards each of which work when the time range is small enough and data can be loaded ‘instantaneously’.

Problem

But when I specify a time range of over an hour, I start to see POST requests getting blocked by browser from the network tab of webpage inspect. This behaviour is consistent across several browsers (Firefox, Edge and Chrome).

The POST requests are blocked almost right as they are sent and don’t fail after a timeout. In Firefox, I see the error NS_BINDING_ABORTED. Grafana dashboard keeps sending the request with no luck (as seen from inspect screenshot below).

xq6xckiI
xq6xckiI1823×577 41.4 KB

After a minute or two, these POST requests intermittently start to succeed out of the blue. I have not been able to figure out what’s causing this issue.

  • What did you expect to happen?
    Grafana dashboard to load graphs for the specified time range.
  • Can you copy/paste the configuration(s) that you are having problems with?

grafana.ini:

[plugin.marcusolsson-csv-datasource]
allow_local_mode = true

[dataproxy]
logging = true

[dashboards]
min_refresh_interval = 1s

[auth]
token_rotation_interval_minutes = 120

[feature_management]
hidden_toggles =
read_only_toggles =
  • Did you receive any errors in the Grafana UI or in related logs? If so, please tell us exactly what they were.
  • Did you follow any online instructions?

I copied the request body and headers of these requests into postman and tried sending from there which worked flawlessly. I also sent the curl request directly to the grafana pod from within the Openshift nodes which also worked without error, leading me to think this may be a grafana issue with browser and not an issue with the load-balancer.

I also exposed grafana on a nodeport service to access it outside of a load-balancer and got the same errors in the browser.

To see if this is an issue with influxdb, I ran the queries specified in the request body directly within influxdb pod and got an appropriate output.

Some online posts suggest firewalls can cause these issues but I don’t have any firewall installed. The client request goes to the load-balancer which routes it to one of the OKD servers.

Another post suggested HTTP traffic to HTTPS connection can be a cause of this. I have self-signed certificates for the HTTPS connections so could this be an issue? So far I haven’t found any evidence backing this in my problem. To negate this issue, I created an HTTP/insecure route and got the same behaviour leading me to think this may also be not the issue.

It appears to me some sort of configuratin issue that would resolve it for the browser. Please let me know how I can further troubleshoot this as I’ve spend sigificant time on this issue without any luck. Please let me know what further info I can provide on this.

Really appreciate any help I can get on this, Thanks

2

What are the errors in the browsers console?
Cors issues, mixed content,…?

3

Hi @jangaraj,

Thanks for your reply. I only see a warning in the console and no errors. Please see below.

image
image1060×140 6.35 KB

4

What’s the url of your Grafana instance? Try to disable all plugins in the browser. How the grafana domain is resolving?Any “antivirus” installed on your machine?

5

@jangaraj my bad, there are actually errors in the console. they just don’t show up right away. Several instances of below 2 errors keep popping up.

image
image1919×624 47.4 KB

As for the url, it is private and only accessible behind a vpn so can’t share it here unfortunately. domain resolves through a private DNS also accessible only from within a VPN.

I have tried opening the dashboard in a fresh browser profile where browser behaves as a new install without any add-ons but the issue was consistent there.

6

Have you recently migrated/upgraded grafana? Or have you recently copied a dashboard JSON from another instance?

7

@yosiasz thanks for your reply. I initially had tig stack on a single linux server. I was using grafana 10.4.0 on that server and that worked just fine. I’ve recently moved TIG stack to Openshift and using grafana 11.3.0 here.

I copied the grafana dashboard JSON from my linux server instance to Openshift instance.

8

Dashboard JSON refer datasource by uid. A datasource uid on one server is different than a datasource uid on another server. In the new server you need to point your dashboard to the proper datasource

These kinds of issues can be avoided if you use provisioning

9

@yosiasz thanks for pointing that out. I’ve fixed the incorrect uids and now I don’t see any errors in the console. However, the dashboard graphs are still not working and I’m still seeing the network errors I shared with the screenshot in the first post. Can you share what could’ve been a cause for that?

10

So you moved something that was working on Linux and now does not work in Open shift

Just like just like the dashboard configuration issue you must have some other configuration issue

11

@yosiasz I copied over grafana.ini and the dashboard config only but will confirm with the colleague too. I have shared grafana.ini already above which is pretty minimal and uses defaults for most options. Does grafana.ini pose any issues?

12

What is the technical reason you copied that over?

13

@yosiasz I wanted to keep grafana behaviour same on both installations and not run into unexpected behaviours like this :sweat_smile: .

1 Like
14

I would recommend you start from scratch, this is a wounded grafana instance :laughing: death by a thousand hacks.

hard to provided support on such a wounded instance of grafana.

15

@yosiasz thanks for the input. Unfortunately I’ve already tried reinstalling several times with no luck. I’ve also tried alternative installation methods for grafana (grafana operator and helm charts) all giving the same result and error :grimacing:

17

Maybe some certificate issues? Ssl?

18

I’ve tried with HTTP only and got the same issues unfortunately :grimacing:.

19

I fixed the issue. min_refresh_interval = 1s was the culprit. linux system had refresh manually turned off that’s why didn’t feel the effect there. Each dashboard request was taking longer than a second and I suspect grafana was sending another identical request each second, leading to the previous one being cancelled by the browser. That also explains why I didn’t get this issue outside of the browser (with postman/curl).

1 Like