How does Grafana attempt to load it's frontend files?

Grafana version 8 (docker image)

I’m going to apologize in advance for not providing source code however the situation I’m seeking help on is on an airgapped system with some fairly stringent rules for moving data of any kind back to internet connected machines. The setup is a bit too complex to even attempt to retype all relevant configurations by hand. Aside, I’m not really looking for help debugging my setup so much as wanting to understand how an aspect of Grafana works so that I can figure out how my setup should be designed in the first place. Hopefully the rest of this matches your template closely enough.

** What are you trying to achieve **
We are trying to setup an instance of Grafana which can access datasources but only for users with the correct credentials.

To elaborate: within the airgapped system I mention there is a second internal system. The internal system is mostly isolated but does have a reverse proxy that can be used to access the services within the system. This reverse proxy also handles authentication to keep access to the internal system restricted. The services within the internal system can communicate with each other freely.

We would like a user facing Grafana that can expose data held by services within the the internal system but only to those who can authenticate with the internal system to begin with. We ran into an issue creating a Grafana instance within the internal system in that we discovered that both the main reverse proxy and Grafana both use a header called ‘Authorization’ to determine access and they are both expecting different things within this header therefore, any communication that can authenticate with the reverse proxy cannot authenticate with Grafana and vice versa. We did look into trying to get Grafana to authenticate in the fashion provided by the main reverse proxy but that line wasn’t looking promising.

** How are you trying to achieve it **

The idea we are currently attempting is to put a second reverse proxy behind the main one and load it with an API key for Grafana. For the users we’d like to show this information we’ll provide them the path to the internal reverse proxy and ideally their authentication will get them through the outer reverse proxy and routed to the inner one which will take their request, swap authorization headers and forward it to Grafana via the internal networks communication and then relay back the dashboards and everything else Grafana serves up.

** What happened **

To some extent this appears to work. The headers get swapped and the API key we provided is validated. Every request we send reaches Grafana and returns with a 200 response so we’re able to retrieve the fonts, stylesheets, icons, and some of the javascript without any difficulty however after taking a bit of time attempting to load it shows the “If you are seeing this Grafana failed to load its application files” error with a list of suggestions. We’re fully willing to believe that either of our proxy configurations might be causing issue but if so we aren’t sure how.

We compared the internal Grafana to a second Grafana instance that we have running outside the internal system and we can see that when one first accesses the external Grafana that all the same requests we saw on the internal Grafana are made but there are several others in addition (mostly more javascript). I presume that these are the application files that Grafana is missing. I am not sure how Grafana can be successfully locating some of it’s files through the various proxies and whatnot but unable to find others that look like they must be in basically the same location (going off the requests).

** What did you expect to happen **

I was hoping (though not exactly expecting) that if the proxies were properly set up to pass the request that users would be able to load the internal Grafana in their browsers and access data like they do the external one.

To restate the main question: How does Grafana attempt to locate it’s frontend files and how does it make the determination that it can’t find them to display that error?

** Did you receive any errors in the Grafana UI or in related logs **

I already mentioned the one shown on the UI. I did not see any errors in the logs for Grafana, or either proxy.

** Did you follow any online instructions **

I have spent quite a bit of time looking in the Grafana documentation both for this current solution and while working on other solutions mentioned throughout this request. The most relevant to this solution would probably be the reverse proxies however I wouldn’t say I was following instructions on those as neither reverse proxy I was working with follows the form of the proxies listed on that page.

As a note: The vast majority of the contstraints of the system exist for a reason and are beyond my control. I do however have full control over the second reverse proxy and so if there is some way the request can be worked to get through I should be able to set it to do so.
–A potential alternate solution would be if there was some way to configure Grafana to use the users authentication when it makes requests to the datasources (so we could configure a datasource to have it look through the main reverse proxy if being accessed by a user with the correct creds) however I couldn’t figure out how to do this while looking through the documentation.

Thanks in advance for the help.

Is there any other information I could provide that would make this easier to answer?

UI “authentification” with API key doesn’t sound very good. API key = API access != UI access => expects a problems with the UI. Your setup doesn’t look good to me.

Generally, you have proxy which is doing authentication so use Auth Proxy | Grafana Labs - your proxy just need to set correct request headers, so Grafana will know user identity from the request. And of course proxy shouldn’t forward authorization header to Grafana, when that authorization header is for the proxy only.

I actually am glad to hear that my setup doesn’t sound good.

With that said, after working with it, the fact that we included an Authorization header doesn’t appear to be the problem or if it is, it isn’t the only one. I changed the proxy to remove the Authorization header completely with no effect on the outcome. As a note, if we don’t remove the Authorization header or set it to something Grafana can accept it refuses to load throwing a ‘Invalid API key’ error. Given the information about API access vs UI access removing it makes more sense than putting in an API key we won’t need but we still need to resolve the issue of the frontend not loading.

This detail likely makes no difference but I figure I’ll mention it just in case. Both Chrome and Edge are willing to navigate to grafana through the two proxy setup (it just fails to load after getting there) however when Firefox attempts the same request it gets no response back at all.

After poking around at it some more, I was able to find some more information which seems more likely to actually be useful. After changing some configurations I do get some console warnings and errors when the page fails to load that I do not get when access the version of the site when it is outside the internal system.

The errors are 'Uncaught SyntaxError: Unexpected token ‘:’. There are several of these but they are all similar. In each case it looks like page content (javascript, css ect) is coming back followed by Cache-Control: public, max-age=31536000 and a few other things that look like headers.The errors only mention the : after Cache-Control though I suspect they wouldn’t like it for Content-Length, Content-Type, Last-Modified, or Date either if the reader had gotten that far.

For anyone noticing this issue in the future the error turned out to be (unsurprisingly) unrelated to Grafana. A logic error in the proxies programming was causing a bit of extra junk to be written into the response which was causing the javascript to error out.