I have written a streaming datasource plugin which creates multiple streams (never ending) http calls (fetch API) to the source through Grafana backend proxy. Everything works fine for 1 or 2 panels added using this plugin. But I observed that when I add multiple panels each with its own set of streaming queries, the Grafana UI gets stuck. Only 2/3 of those queries return while others get stuck in Pending mode.
Moreover if I try to connect to Grafana UI in a different browser window that gets stuck as well. If I close my previous browser window, the connection succeeds immediately.
I increase the max_idle_conn and max_open_connto 20 but the issues still persists.
Please help.
I found the root cause of the issue. It has nothing to do with Gragana.
Every browser has a restriction of max 6 concurrent fetch (or XHR) requests per domain. In my case all the connections are open ended meaning the request never finishes, a very common use case for streaming. Because of this restriction only 6 reality time charts were showing up. Once these 6 connections are exhausted even other functions of Grafana e.g. Save dashboard will not work as that is another XHR to same domain.
So to say there is a limitation that in case of streaming datasource only 5 real time panels will run at a time in one browser session across tabs.
Now I am exploring the websocket based approach.
1 Like
Web socket worked like a charm. Just that I had to setup a separate middleware node server to which my plugin is communicating. I had to set Access method “browser” as there is no support for web sockets in Grafana backend server. Now I can create as many connections as I want with the backend as there is no restriction on number of connections for web sockets unlike XHR.
It would be great if Grafana also gives web socket proxy along with http proxy. That will make things even simpler.
Can you give any details on how you got it to work … i guess I need to do the same …
