Nginx, reverse-proxy subpath failing some requests fails with error 404

Grafana Configuration
,
1

Hi
I’m trying to setup nginx and Grafana with reverse proxy and subpath.
Grafana version: v11.1.0
I have followed the guide from here: Run Grafana behind a reverse proxy | Grafana Labs
After i login to grafana i see this:

Error
<html> 
<head>
<title>404 Not Found</title>
</head>
 <body> 
<center>
<h1>404 Not Found</h1>
</center> 
<hr><center>
nginx</center> 
</body>
 </html> 
<!-- a padding to disable MSIE and Chrome friendly error page -->
 <!-- a padding to disable MSIE and Chrome friendly error page -->
 <!-- a padding to disable MSIE and Chrome friendly error page --> 
<!-- a padding to disable MSIE and Chrome friendly error page --> 
<!-- a padding to disable MSIE and Chrome friendly error page --> 
<!-- a padding to disable MSIE and Chrome friendly error page -->

If i inspect the Grafana site i see this:

Request URL:
https:// test.domain.com/grafana/public/build/DashboardPageProxy.900c36a42804a5ac4e00.js
Request Method:
GET
Status Code:
200 OK
Remote Address:
10.10.198.63:443
Referrer Policy:
strict-origin-when-cross-origin

Request URL:
https:// test.domain.com/grafana/api/dashboards/home
Request Method:
GET
Status Code:
404 Not Found
Remote Address:
10.10.198.63:443
Referrer Policy:
strict-origin-when-cross-origin

I have 2 nginx config files that look like this:
/etc/nginx/nginx.conf

user www-data;
worker_processes  auto;
error_log  /var/log/nginx/error.log notice;
pid        /var/run/nginx.pid;
events {
    worker_connections  1024;
}
http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';
    access_log  /var/log/nginx/access.log  main;
    sendfile        on;
    #tcp_nopush     on;
    keepalive_timeout  65;
    #gzip  on;
    include /etc/nginx/conf.d/*.conf;
}
/etc/nginx/conf.d/Zabbix.conf
# This is required to proxy Grafana Live WebSocket connections.
map $http_upgrade $connection_upgrade {
  default upgrade;
  '' close;
}
upstream grafana {
  server localhost:3000;
}
server {
    listen 80;
    server_name test.domain.com;
    return 301 https://$host$request_uri;
}
server {
    listen 443 quic reuseport;
    listen [::]:443 quic reuseport;
    listen 443 ssl;
    listen [::]:443 ssl;
    server_name test.domain.com;
    ssl_certificate /etc/letsencrypt/live/test.domain.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/test.domain.com/privkey.pem;
    ssl_dhparam /etc/nginx/ssl/dhparam.pem;
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_prefer_server_ciphers on;
    ssl_ciphers 'TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305';
    gzip off;
    server_tokens off;
#    add_header X-Frame-Options "DENY" always;
#    add_header X-Content-Type-Options "nosniff" always;
#    add_header X-XSS-Protection "1; mode=block" always;
#    add_header Strict-Transport-Security "max-age=31536000; includeSubDomains; preload" always;
#    add_header Referrer-Policy "no-referrer-when-downgrade" always;
#    add_header Content-Security-Policy "default-src 'self'; script-src 'self' 'unsafe-inline'; style-src 'self' 'unsafe-inline'; object-src 'none'; base-uri 'self'; form-action 'self'" always;
    root /usr/share/zabbix;
    index index.php;
    location = /favicon.ico {
        log_not_found off;
    }
    location / {
        try_files $uri $uri/ =404;
    }
    location /assets {
        access_log off;
        expires 10d;
    }
    location ~ /\.ht {
        deny all;
    }
    location ~ /(api\/|conf[^\.]|include|locale) {
        deny all;
        return 404;
    }
    location /vendor {
        deny all;
        return 404;
    }
    location ~ [^/]\.php(/|$) {
        fastcgi_pass unix:/var/run/php/zabbix.sock;
        fastcgi_split_path_info ^(.+\.php)(/.+)$;
        fastcgi_index index.php;
        fastcgi_param DOCUMENT_ROOT /usr/share/zabbix;
        fastcgi_param SCRIPT_FILENAME /usr/share/zabbix$fastcgi_script_name;
        fastcgi_param PATH_TRANSLATED /usr/share/zabbix$fastcgi_script_name;
        include fastcgi_params;
        fastcgi_param QUERY_STRING $query_string;
        fastcgi_param REQUEST_METHOD $request_method;
        fastcgi_param CONTENT_TYPE $content_type;
        fastcgi_param CONTENT_LENGTH $content_length;
        fastcgi_intercept_errors on;
        fastcgi_ignore_client_abort off;
        fastcgi_connect_timeout 60;
        fastcgi_send_timeout 180;
        fastcgi_read_timeout 180;
        fastcgi_buffer_size 128k;
        fastcgi_buffers 4 256k;
        fastcgi_busy_buffers_size 256k;
        fastcgi_temp_file_write_size 256k;
    }
    location /grafana/ {
        proxy_set_header Host $host;
        proxy_pass http://grafana;
#       rewrite  ^/grafana/(.*)  /$1 break;
    }
    # Proxy Grafana Live WebSocket connections.
    location /grafana/api/live/ {
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
        proxy_set_header Host $host;
        proxy_pass http://grafana;
    }
    location /nginx_status {
        stub_status;
        allow 127.0.0.1;
        deny all;
    }
}

My Grafana file /etc/grafana/grafana.ini

# The public facing domain name used to access grafana from a browser
domain = test.domain.com

# The full public facing url you use in browser, used for redirects and emails
# If you use reverse proxy and sub path specify full url (with sub path)
root_url = %(protocol)s://%(domain)s:%(http_port)s/grafana/
# Serve Grafana from subpath specified in `root_url` setting. By default it is set to `false` for compatibility reasons.
serve_from_sub_path = true

If i look at the nginx log i see this:

2024/07/19 08:49:56 [warn] 12593#12593: *8243 an upstream response is buffered to a temporary file /var/cache/nginx/proxy_temp/6/13/0000000136 while reading upstream, client: 10.0.60.12, server: test.domain.com, request: "GET /grafana/public/build/9778.aeed2343301c3e5780fc.js HTTP/1.1", upstream: "http:// 127.0.0.1:3000/grafana/public/build/9778.aeed2343301c3e5780fc.js", host: "test.domain.com", referrer: "https://test.domain.com/grafana/"
2024/07/19 08:49:56 [warn] 12590#12590: *8103 an upstream response is buffered to a temporary file /var/cache/nginx/proxy_temp/8/13/0000000138 while reading upstream, client: 10.0.60.12, server: test.domain.com, request: "GET /grafana/public/build/1247.34022e0fa8ad25da7b32.js HTTP/1.1", upstream: "http:// 127.0.0.1:3000/grafana/public/build/1247.34022e0fa8ad25da7b32.js", host: "test.domain.com", referrer: "https://test.domain.com/grafana/"
2024/07/19 08:49:56 [warn] 12591#12591: *8241 an upstream response is buffered to a temporary file /var/cache/nginx/proxy_temp/7/13/0000000137 while reading upstream, client: 10.0.60.12, server: test.domain.com, request: "GET /grafana/public/build/app.8222b35d5a14e3d0ea28.js HTTP/1.1", upstream: "http:// 127.0.0.1:3000/grafana/public/build/app.8222b35d5a14e3d0ea28.js", host: "test.domain.com", referrer: https://test.domain.com/grafana/
2

I know this post is over a year old, but it was one of the first things that came up on a search for me when I was having the same problem. I too am hosting Zabbix on the system and am adding Grafana to it, and wanted it to run from a sub-path (/grafana/).

I resorted to talking to Gemini AI and it was actually able to help, so I figured I’d share it here.

The problem is this section, which is for “security” and part of the default Zabbix nginx config:

location ~ /(api\/|conf[^\.]|include|locale) {
        deny all;
        return 404;
}

This is a fuzzy URL match against these strings, and it breaks parts of Grafana in subtle ways. :slight_smile:

The fix is to modify your Grafana-related location directives and add the magic incantation of ^~.

So using your original version, you want them to look like this:

location ^~ /grafana/ {
        proxy_set_header Host $host;
        proxy_pass http://grafana;
        # KEEP THE REWRITE COMMENTED OUT!
        # grafana's "serve_from_sub_path=true" makes it unnecessary
        #rewrite  ^/grafana/(.*)  /$1 break;
    }
    # Proxy Grafana Live WebSocket connections.
    location ^~ /grafana/api/live/ {
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
        proxy_set_header Host $host;
        proxy_pass http://grafana;
    }

Those two (four) extra characters will solve 99% of your problems… They cause nginx to match these directives before the fuzzy ~ regex ones.

The other thing you will probably want to do - in grafana.ini, do NOT set protocol=https. Leave it at the default http, and just edit the root_url to have a static “https://” at the beginning (instead of the “variable” of %(protocol)s. Changing “protocol” has deeper meaning to Grafana (you’re telling it that it’s supposed to be doing SSL/TLS, when it actually isn’t if you have nginx as a proxy in front).

I hope that this helps someone out in the future as I wasted a ton of time spinning my wheels on this before resorting to AI and actually getting a helpful answer from it. (I was mildly surprised!)