Grafana UI gets stuck when multiple "fetch" queries are fired

Development
#1

I have written a streaming datasource plugin which creates multiple streams (never ending) http calls (fetch API) to the source through Grafana backend proxy. Everything works fine for 1 or 2 panels added using this plugin. But I observed that when I add multiple panels each with its own set of streaming queries, the Grafana UI gets stuck. Only 2/3 of those queries return while others get stuck in Pending mode.

Moreover if I try to connect to Grafana UI in a different browser window that gets stuck as well. If I close my previous browser window, the connection succeeds immediately.

I increase the max_idle_conn and max_open_connto 20 but the issues still persists.

Please help.

#2

I found the root cause of the issue. It has nothing to do with Gragana.

Every browser has a restriction of max 6 concurrent fetch (or XHR) requests per domain. In my case all the connections are open ended meaning the request never finishes, a very common use case for streaming. Because of this restriction only 6 reality time charts were showing up. Once these 6 connections are exhausted even other functions of Grafana e.g. Save dashboard will not work as that is another XHR to same domain.

So to say there is a limitation that in case of streaming datasource only 5 real time panels will run at a time in one browser session across tabs.

Now I am exploring the websocket based approach.

1 Like