Facing issue while Configuring Grafana behind a reverse proxy

Hi Team

While configuring Grafana behind a reverse proxy. I am getting 404. I have shared below my changes in configuration file. Please do the needful where am missing. Appreciate your valuable input on this,

Grafana Configuration File[server]

protocol = http

http_addr =

http_port = 3000

domain = example.com

enforce_domain = false

root_url = %(protocol)s://%(domain)s:%(http_port)s/grafana/

serve_from_sub_path = true

NGINX Configuration File
http {
include mime.types;
default_type application/octet-stream;

map $http_upgrade $connection_upgrade {
default upgrade;
'' close;
}

upstream grafana {
server community.grafana.com;
}

#log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
#                  '$status $body_bytes_sent "$http_referer" '
#                  '"$http_user_agent" "$http_x_forwarded_for"';

#access_log  logs/access.log  main;

sendfile        on;
#tcp_nopush     on;

#keepalive_timeout  0;
keepalive_timeout  65;

#gzip  on;

server {
    listen       80;
    #server_name  localhost;
	
	root /usr/share/nginx/html;
	index index.html index.htm;

	location / {
	proxy_set_header Host $http_host;
	proxy_pass http://grafana;
	}

	# Proxy Grafana Live WebSocket connections.
	location /api/live/ {
	proxy_http_version 1.1;
	proxy_set_header Upgrade $http_upgrade;
	proxy_set_header Connection $connection_upgrade;
	proxy_set_header Host $http_host;
	proxy_pass http://grafana;
	}

Getting a 404 https://www.example.com/grafana

1 Like

Hey Rohit, as mentioned in your nginx config,

You’re proxying requests at “/” to Grafana running at 3000 port.

But in your Grafana config, you’ve mentioned that root URL will be example.com/grafana.

In your nginx config, the /grafana path seem to be missing. Could you please add the below, reload nginx and try to visit the URL ?

  location /grafana/ {
    rewrite  ^/grafana/(.*)  /$1 break;
    proxy_set_header Host $http_host; 
    proxy_pass http://grafana;
  }

More on this is perfectly documented here : Run Grafana behind a reverse proxy | Grafana Labs

Hope your issue gets resolved :slight_smile:

Hi Tanmay

As said, i added the same line, But still getting the same error.

NGINX Configuration File
http {
include mime.types;
default_type application/octet-stream;

map $http_upgrade $connection_upgrade {
default upgrade;
'' close;
}

upstream grafana {
server community.grafana.com;
}

#log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
#                  '$status $body_bytes_sent "$http_referer" '
#                  '"$http_user_agent" "$http_x_forwarded_for"';

#access_log  logs/access.log  main;

sendfile        on;
#tcp_nopush     on;

#keepalive_timeout  0;
keepalive_timeout  65;

#gzip  on;

server {
    listen       80;
    #server_name  localhost;
	
	root /usr/share/nginx/html;
	index index.html index.htm;

	location /grafana/ {
	rewrite  ^/grafana/(.*)  /$1 break;
	proxy_set_header Host $http_host; 
	proxy_pass http://grafana;
	}

	# Proxy Grafana Live WebSocket connections.
	location /api/live/ {
	proxy_http_version 1.1;
	proxy_set_header Upgrade $http_upgrade;
	proxy_set_header Connection $connection_upgrade;
	proxy_set_header Host $http_host;
	proxy_pass http://grafana;
	}

I tried this setup on my local, and I made it working :

Here’s my nginx config :


events {
    worker_connections   1000;
}

http {
map $http_upgrade $connection_upgrade {
default upgrade;
'' close;
}

upstream grafana {
  server grafana:3000;
}

server {
  listen 80;
  root /usr/share/nginx/www;
  index index.html index.htm;

  location /grafana/ {
    rewrite  ^/grafana/(.*)  /$1 break;
    proxy_set_header Host $http_host; 
    proxy_pass http://grafana;
  }

  # Proxy Grafana Live WebSocket connections.
  location /grafana/api/live/ {
    rewrite  ^/grafana/(.*)  /$1 break;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection $connection_upgrade;
    proxy_set_header Host $http_host;
    proxy_pass http://grafana;
  }
}
}

My grafana config :

[server]
domain = example.com
root_url = %(protocol)s://%(domain)s:%(http_port)s/grafana/
serve_from_sub_path = true

Don’t mind the grafana URL to be grafana instead of localhost since I’m running both nginx and grafana in docker.

Can you validate if grafana is running at 3000 ?

And please cross validate the config I shared with yours.

What happens if you try to serve Grafana at root URL i.e “/”, does it work ?

Hi Tanmay

Thanks for helps.

I am running both grafana & NGINX locally, Is this causing a problem.
I have validated grafana url http://grafana.staged-by-discourse.com/grafana/login, Grafana is running properly.
But after configuring the reverse proxy, https://example.com/grafana. I am getting 404 error.

Grafana Config [Am doing changes in defaults.ini & sample.ini]

The public facing domain name used to access grafana from a browser

#domain = localhost
domain = example.com

Redirect to correct domain if host header does not match domain

Prevents DNS rebinding attacks

enforce_domain = false

The full public facing url

root_url = %(protocol)s://%(domain)s:%(http_port)s/grafana/

Serve Grafana from subpath specified in root_url setting. By default it is set to false for compatibility reasons.

serve_from_sub_path = true

nginx config [ server community.grafana.com; I added localhost because grafana is running locally.]
http {
include mime.types;
default_type application/octet-stream;

map $http_upgrade $connection_upgrade {
default upgrade;
'' close;
}

upstream grafana {
server community.grafana.com;
}

#log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
#                  '$status $body_bytes_sent "$http_referer" '
#                  '"$http_user_agent" "$http_x_forwarded_for"';

#access_log  logs/access.log  main;

sendfile        on;
#tcp_nopush     on;

#keepalive_timeout  0;
keepalive_timeout  65;

#gzip  on;

server {
	listen 80;
	root /usr/share/nginx/www;
	index index.html index.htm;

	location /grafana/ {
	rewrite  ^/grafana/(.*)  /$1 break;
	proxy_set_header Host $http_host; 
	proxy_pass http://grafana;
	}

    #charset koi8-r;

    #access_log  logs/host.access.log  main;

	# Proxy Grafana Live WebSocket connections.
	location /grafana/api/live/ {
	rewrite  ^/grafana/(.*)  /$1 break;
	proxy_http_version 1.1;
	proxy_set_header Upgrade $http_upgrade;
	proxy_set_header Connection $connection_upgrade;
	proxy_set_header Host $http_host;
	proxy_pass http://grafana;
	}

Even i created a new grafana.ini and perform the same changes its not working with reverse proxy configuration.

I restarted grafana as well.

Hey Rohit, here’s what I’ve done. You can cross verify the below with your config to find out the bug, since the config you shared is not complete or not formatted. Its difficult to read.

First, I’m assuming you’re running running this in Linux. A complete guide on how to configure this is written here.

I ran my setup in Ubuntu.

So for that I’ll edit the defaults.ini.

From doc :

If you installed Grafana using the deb or rpm packages, then your configuration file is located at /etc/grafana/grafana.ini and a separate custom.ini is not used.

Next, I’ve edited the doc for only 3 parameters I’ve shared above.

Here’s my config :

# The http port to use
http_port = 3000

# The public facing domain name used to access grafana from a browser
domain = example.com

# Redirect to correct domain if host header does not match domain
# Prevents DNS rebinding attacks
enforce_domain = false

# The full public facing url
root_url = %(protocol)s://%(domain)s:%(http_port)s/grafana/

# Serve Grafana from subpath specified in `root_url` setting. By default it is set to `false` for compatibility reasons.
serve_from_sub_path = true

#....other default config lines

Next, lets look at the nginx config file :

events {
    worker_connections  1024;
}

http {
include mime.types;
default_type application/octet-stream;

map $http_upgrade $connection_upgrade {
default upgrade;
'' close;
}

upstream grafana {
server community.grafana.com;
}

#log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
#                  '$status $body_bytes_sent "$http_referer" '
#                  '"$http_user_agent" "$http_x_forwarded_for"';

#access_log  logs/access.log  main;

sendfile        on;
#tcp_nopush     on;

#keepalive_timeout  0;
keepalive_timeout  65;

#gzip  on;

server {
        listen 80;
        root /usr/share/nginx/www;
        index index.html index.htm;

        location /grafana/ {
        rewrite  ^/grafana/(.*)  /$1 break;
        proxy_set_header Host $http_host;
        proxy_pass http://grafana;
        }

    #charset koi8-r;

    #access_log  logs/host.access.log  main;

        # Proxy Grafana Live WebSocket connections.
        location /grafana/api/live/ {
        rewrite  ^/grafana/(.*)  /$1 break;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
        proxy_set_header Host $http_host;
        proxy_pass http://grafana;
        }
}
}

I did a diff of your file and my file and :

diff nginx.conf rohit.conf
1,3c1
< events {
<     worker_connections  1024;
< }
---
>
8d5
<
56,57d52
< }
< }

Only difference between my file and your file is :

  1. events section is missing
  2. Two brackets at the end is missing in your file.

I’m assuming you must be familiar with nginx. Here’s how you can test your nginx file for valid syntax :

nginx -t

Which should give you :

nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

Once that’s valid, I added my hosts file with below entry :

127.0.0.1 example.com

And I can visit the web now :

Note : if you’re running a Red Hat flavoured system, additional config is needed for reverse proxy.

Documented here : Install on RPM-based Linux | Grafana documentation

1 Like

Hi Tanmay

Thanks you so much, am able to set up Reverse Proxy successfully and its running successfully.

One last question while embeding grafana iframe in angular with authentication token in header.
a) In proxy server, How to pass credentials for viever into request headers?

Am getting a CORS issues while accessing the url into angular application.
Error: Access to XMLHttpRequest at ‘http://example.com/grafana/d/ZmqS29WVk/ade?orgId=1&from=now-2y&to=now&random=1661940842000Wed%20Aug%2031%202022%2015:44:02%20GMT+0530%20(India%20Standard%20Time)’ from origin ‘http://localhost:4200’ has been blocked by CORS policy: Response to preflight request doesn’t pass access control check: Redirect is not allowed for a preflight request.

1 Like

this might help;

csrf_trusted_origins=example.com