Getting Grafana "origin not allowed" error, with 403 in HTTP archive logs

We have Grafana installed on AKS cluster. The Grafana instance is exposed publicly and we also have internal URL for Grafana. The Grafana Traffic comes and hits Application Gateway that has internal IP address of the Grafana ingress as a Backendpool. But this configuration gives 403 unauthorized error while trying to invite new user. But the invitation works on the internal Grafana URL, when browsed through a Virtual machine which is present in the same Vnet as the AKS cluster. Only ingress for Grafana is implemented without reverse proxy. Is it mandatory for us to use reverse proxy along with Grafana ingress for this config to work?

You can have whatever (nginx, ALB, NLB, API GW, LB, reverse proxy, …) you want in front of Grafan. But Grafana must receive correct request Host header, which is matching domain configured in root_url in grafana.ini.

Documentation how to achieve that for some popular reverse proxies: Run Grafana behind a reverse proxy | Grafana Labs

Thanks for your reply! root url is pointing to our internal grafana URL, which seems to be fine, but the requests are being made on the public URL. Can you please provide your email address so that, I can be in touch with you directly?

Sorry, this is community forum: How to use the Grafana forums

If you need “email support”, then I would recommend some paid Grafana Lab’s support/service, where you will have included support.

Thank you! We are using Grafana free version and its not Enterprise one. Do you do freelancing?

We tried all permutations and combinations with configuration of Grafana, but the issue still persists. The root_url is configured as the internal URL of Grafana as per grafana.ini file in the pod. But still the issue persists.

How many permutations and combinations is there possible? I think it is really high number.

Not many though :slight_smile: We have annotations section that has cors settings are present: “true”“URLs of internet and internal facing entries”. However, in grafana.ini we dont see these cors settings not sure why. Also the log location /var/log/grafana is empty. Nothing is registered there. In production instance we dont have this problem as both internal facing and internet facing URLs are one and the same.

This is not a Grafana config. That’s K8S ingress config. But you don’t face CORS issue, so why you are configuring CORS then?

Thats right.Grafana has ingress and thats how we access it, it does not work with or without CORS. I just mentioned the kind of config we have. Also wile trying to invite users from internet facing portal we got 403 unauthorized error. But this error is persistent.

Let’s think:
1.) How many domains you can have in root_url?
2.) How many domains you need to serve?

In the root URL, there is only one that is pointing to internal URL. There is only one parent domain and two child domains, one configured for internet access via CDN and the other configured as ingress of grafana on the aks cluster. The traffic comes from internet facing URL to Application gateway that has internal URL as the backend pool, so at last the traffic is served by internal URL.

So how can you achieve that Grafana will receive only the domain, which is configured in root_url?

Correct, when we invite the user or do operations in the internet URL, the traffic ultimately reaches the internal URL and that it should serve the requests. The login of the internet facing URL works and later, it does not allow us to do any operations and each operation fails with “Origin not allowed” error.