Hi all,
I am sorry for the 100th topic regarding this, but I haven’t found the solution so far.
What?
I can access the page without problem with http://my_server_ip:3000 but https://grafana.my_domain.com just returns error 403.
Symptoms?
I run Grafana as a docker on my local server and use nginx (swag docker) as reverse proxy.
Each time I try to access grafana through the reverse proxy I get an error 403 but don’t see any hinting log to see what is wrong. The grafana server is running in a docker container named grafana. The ssl handling works as other services have no problem with it. I also added the proxy_set_header Host $http_host; line as I saw it created problems but it didn’t solve it.
Anyone can see what is the problem in the nginx configuration?
Nginx proxy-conf:
server {
listen 443 ssl;
listen [::]:443 ssl;
server_name grafana.*;
include /config/nginx/ssl.conf;
client_max_body_size 0;
location / {
include /config/nginx/proxy.conf;
include /config/nginx/resolver.conf;
proxy_set_header Host $http_host;
set $upstream_app grafana;
set $upstream_port 3000;
set $upstream_proto http;
proxy_pass $upstream_proto://$upstream_app:$upstream_port;
}
location ~ (/grafana)?/api {
include /config/nginx/proxy.conf;
include /config/nginx/resolver.conf;
set $upstream_app grafana;
set $upstream_port 3000;
set $upstream_proto http;
proxy_pass $upstream_proto://$upstream_app:$upstream_port;
}
location ~ (/grafana)?/metrics {
include /config/nginx/proxy.conf;
include /config/nginx/resolver.conf;
set $upstream_app grafana;
set $upstream_port 3000;
set $upstream_proto http;
proxy_pass $upstream_proto://$upstream_app:$upstream_port;
}
}
Hi,
try to put
set $upstream_app grafana; using your IP
set $upstream_app 192.168.1.100; for example.
Regards,
Fadjar
THe IP isn’t the problem as the container can find it (it is handled by docker network).
If nginx couldn’t reach the grafana container I would get a log entry saying it can’t find it, plus if I ping fron nginx to grafana (using name not IP) it responds. So I don’t think the problem comes from that.
Is there any firewall running to block port 3000 at the target?
No firewall, I use the official grafana image (Docker).
I can access grafana locally with grafana:3000, only grafana.mydns.com doesn’t work so it seems to be something fishy in the nginx configuration.
I hope you got through this setup?
Based on the provided configuration, the issue may also related to the server_name directive. It is set to grafana.* , which will match any subdomain of the grafana domain, but not the grafana domain itself. To fix this, you should change grafana.* to grafana.my_domain.com.
The HTTP 403 Forbidden response status code indicates that the server understands the request but refuses to authorize it. In case of FW or any other unavailability I would expect a 5xx code.
1 Like
I will run through the link make sure everything is fine. Be right back. Thanks @jiriwetter for the link.
Hi again @jiriwetter
Unfortunately the example doesn’t fix it. If I copy the given example it redirects to index.html (as I suppose it can’t find grafana on the localhost as it is running in a different container). If I replace localhost by grafana (as that is the IP of the container) then I still get the error 403.
Are there any logs I can check to find more hints of what goes wrong between nginx and grafana?
I can’t see any entry in the grafana containers and the nginx log is also not helping much.
The only things I see are in nginx logs, I can see the access at 07:21:15:
...
138.199.xx.xx - - [03/May/2023:07:21:15 +0200] "GET / HTTP/2.0" 403 146 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:109.0) Gecko/20100101 Firefox/112.0"
But nothing in error.log:
...
2023/05/03 07:18:04 [alert] 527#527: [lua] init_by_lua:8: [Crowdsec] Initialisation done
Please try to disable Crowdsec first, I think there’s a problem with Crowsec lua scripts.
For testing better using vanilla nginx without mod_sec or other modules, such as resty or Crowdsec.
If the nginx vanilla distribution works, add one by one the hardening scripts…
Regards,
Fadjar
Hi @kodsama,
there are a lot of places to make mistakes. I did the same thing you’re trying to do. So I have a proxy server where I have SSL termination that redirects all requests to the docker host where the Grafana container is running.
Internet > proxy server > docker host > container
I’m using this simplest possible configuration so that nothing else confuses you.
server {
listen 80;
server_name grafana.xx.cz;
return 301 https://grafana.xx.cz$request_uri;
}
server {
server_name grafana.xx.cz;
listen 443 ssl http2;
listen [::]:443 ssl http2;
ssl on;
ssl_certificate /etc/letsencrypt/live/grafana.xx.cz/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/grafana.xx.cz/privkey.pem;
index index.html;
location / {
proxy_pass http://10.52.4.14:3000/;
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto "https";
}
}
$ curl -sk -IL https://grafana.xx.cz
HTTP/2 302
server: nginx/1.18.0
date: Wed, 03 May 2023 06:51:39 GMT
content-type: text/html; charset=utf-8
cache-control: no-cache
expires: -1
location: /login
pragma: no-cache
set-cookie: redirect_to=%2F; Path=/; HttpOnly; SameSite=Lax
x-content-type-options: nosniff
x-frame-options: deny
x-xss-protection: 1; mode=block
HTTP/2 200
server: nginx/1.18.0
date: Wed, 03 May 2023 06:51:39 GMT
content-type: text/html; charset=UTF-8
cache-control: no-cache
expires: -1
pragma: no-cache
x-content-type-options: nosniff
x-frame-options: deny
x-xss-protection: 1; mode=block
I think you have a mistake on the nginx side. But it’s very hard to advise when we don’t know what other settings you are using besides nginx - maybe on the OS side. Anyway, the higher logging level is described here:
http://nginx.org/en/docs/debugging_log.html
2 Likes
Yes, it works now!
There was something fishy which was part of the
include /config/nginx/proxy.conf;
line of the default grafana nginx config. Commenting it out and adding the extra parameters like you had fixed it 
Thanks for the great support!
