Unexpected error behind reverse proxy

Hi guys! I’m trying to setup grafana behind nginx reverse proxy. Configuration is as follows:

nginx.conf:

    server {
      listen 80;
      root /usr/share/nginx/www;
      index index.html index.htm;
      server_name grafana.domain.com;
      location / {
       proxy_pass http://localhost:3000/;
        }
    }

grafana.ini

[server]
domain = grafana.domain.com

If l’m trying to change my password/add user to org, give user admin rights iI’m getting

Possibly unhandled rejection: {"message":"Unexpected error"} [angular.js:14700](webpack:///node_modules/angular/angular.js)

Possibly unhandled rejection: {"data":null,"status":-1,"config":{"method":"PUT","transformRequest":[null],"transformResponse":[null],"jsonpCallbackParam":"callback","url":"api/admin/users/2/permissions","data":{"isGrafanaAdmin":true},"retry":0,"headers":{"X-Grafana-Org-Id":1,"Accept":"application/json, text/plain, */*","Content-Type":"application/json;charset=utf-8"}},"statusText":"","xhrStatus":"error"}
    ​

I’m struggling with it second day already (I tried to chage headers, unix socket instead of http port, with and without subpath ) but still no luck. without reverse proxy it works completely fine. Any ideas?

nginx’s error logs in debug mode looks like this:

2019/10/25 10:36:08 [debug] 16329#16329: *450 write new buf t:1 f:0 00005571295AEF10, pos 00005571295AEF10, size: 353 file: 0, size: 0
2019/10/25 10:36:08 [debug] 16329#16329: *450 http write filter: l:0 f:0 s:353
2019/10/25 10:36:08 [debug] 16329#16329: *450 http cacheable: 0
2019/10/25 10:36:08 [debug] 16329#16329: *450 http proxy filter init s:200 h:0 c:0 l:23
2019/10/25 10:36:08 [debug] 16329#16329: *450 http upstream process upstream
2019/10/25 10:36:08 [debug] 16329#16329: *450 pipe read upstream: 0
2019/10/25 10:36:08 [debug] 16329#16329: *450 pipe preread: 23
2019/10/25 10:36:08 [debug] 16329#16329: *450 pipe buf free s:0 t:1 f:0 0000557129598470, pos 000055712959859A, size: 23 file: 0, size: 0
2019/10/25 10:36:08 [debug] 16329#16329: *450 pipe length: 23
2019/10/25 10:36:08 [debug] 16329#16329: *450 input buf #0
2019/10/25 10:36:08 [debug] 16329#16329: *450 pipe write downstream: 1
2019/10/25 10:36:08 [debug] 16329#16329: *450 pipe write downstream flush in
2019/10/25 10:36:08 [debug] 16329#16329: *450 http output filter "/login?"
2019/10/25 10:36:08 [debug] 16329#16329: *450 http copy filter: "/login?"
2019/10/25 10:36:08 [debug] 16329#16329: *450 http postpone filter "/login?" 0000557129598008
2019/10/25 10:36:08 [debug] 16329#16329: *450 write old buf t:1 f:0 00005571295AEF10, pos 00005571295AEF10, size: 353 file: 0, size: 0
2019/10/25 10:36:08 [debug] 16329#16329: *450 write new buf t:1 f:0 0000557129598470, pos 000055712959859A, size: 23 file: 0, size: 0
2019/10/25 10:36:08 [debug] 16329#16329: *450 http write filter: l:0 f:0 s:376
2019/10/25 10:36:08 [debug] 16329#16329: *450 http copy filter: 0 "/login?"
2019/10/25 10:36:08 [debug] 16329#16329: *450 pipe write downstream done
2019/10/25 10:36:08 [debug] 16329#16329: *450 event timer: 8, old: 7308247, new: 7308275
2019/10/25 10:36:08 [debug] 16329#16329: *450 http upstream exit: 0000000000000000
2019/10/25 10:36:08 [debug] 16329#16329: *450 finalize http upstream request: 0
2019/10/25 10:36:08 [debug] 16329#16329: *450 finalize http proxy request
2019/10/25 10:36:08 [debug] 16329#16329: *450 free rr peer 1 0
2019/10/25 10:36:08 [debug] 16329#16329: *450 close http upstream connection: 8
2019/10/25 10:36:08 [debug] 16329#16329: *450 free: 0000557129574690, unused: 48
2019/10/25 10:36:08 [debug] 16329#16329: *450 event timer del: 8: 7308247
2019/10/25 10:36:08 [debug] 16329#16329: *450 reusable connection: 0
2019/10/25 10:36:08 [debug] 16329#16329: *450 http upstream temp fd: -1
2019/10/25 10:36:08 [debug] 16329#16329: *450 http output filter "/login?"
2019/10/25 10:36:08 [debug] 16329#16329: *450 http copy filter: "/login?"
2019/10/25 10:36:08 [debug] 16329#16329: *450 http postpone filter "/login?" 00007FFF190B4900
2019/10/25 10:36:08 [debug] 16329#16329: *450 write old buf t:1 f:0 00005571295AEF10, pos 00005571295AEF10, size: 353 file: 0, size: 0
2019/10/25 10:36:08 [debug] 16329#16329: *450 write old buf t:1 f:0 0000557129598470, pos 000055712959859A, size: 23 file: 0, size: 0
2019/10/25 10:36:08 [debug] 16329#16329: *450 write new buf t:0 f:0 0000000000000000, pos 0000000000000000, size: 0 file: 0, size: 0
2019/10/25 10:36:08 [debug] 16329#16329: *450 http write filter: l:1 f:0 s:376
2019/10/25 10:36:08 [debug] 16329#16329: *450 http write filter limit 0
2019/10/25 10:36:08 [debug] 16329#16329: *450 writev: 376 of 376
2019/10/25 10:36:08 [debug] 16329#16329: *450 http write filter 0000000000000000
2019/10/25 10:36:08 [debug] 16329#16329: *450 http copy filter: 0 "/login?"
2019/10/25 10:36:08 [debug] 16329#16329: *450 http finalize request: 0, "/login?" a:1, c:1
2019/10/25 10:36:08 [debug] 16329#16329: *450 set http keepalive handler
2019/10/25 10:36:08 [debug] 16329#16329: *450 http close request
2019/10/25 10:36:08 [debug] 16329#16329: *450 http log handler
2019/10/25 10:36:08 [debug] 16329#16329: *450 free: 0000557129598470
2019/10/25 10:36:08 [debug] 16329#16329: *450 free: 0000557129590960, unused: 2
2019/10/25 10:36:08 [debug] 16329#16329: *450 free: 0000557129597460, unused: 16
2019/10/25 10:36:08 [debug] 16329#16329: *450 free: 00005571295AED70, unused: 2604
2019/10/25 10:36:08 [debug] 16329#16329: *450 free: 0000557129599690
2019/10/25 10:36:08 [debug] 16329#16329: *450 hc free: 0000000000000000
2019/10/25 10:36:08 [debug] 16329#16329: *450 hc busy: 0000000000000000 0
2019/10/25 10:36:08 [debug] 16329#16329: *450 tcp_nodelay
2019/10/25 10:36:08 [debug] 16329#16329: *450 reusable connection: 1
2019/10/25 10:36:08 [debug] 16329#16329: *450 event timer add: 3: 75000:7323275
2019/10/25 10:36:08 [debug] 16329#16329: timer delta: 28
2019/10/25 10:36:08 [debug] 16329#16329: worker cycle
2019/10/25 10:36:08 [debug] 16329#16329: epoll timer: 75000
2019/10/25 10:36:21 [debug] 16329#16329: epoll: fd:6 ev:0001 d:00007F8FFA328010
2019/10/25 10:36:21 [debug] 16329#16329: accept on 0.0.0.0:80, ready: 0
2019/10/25 10:36:21 [debug] 16329#16329: posix_memalign: 0000557129594B60:512 @16
2019/10/25 10:36:21 [debug] 16329#16329: *452 accept: 94.140.208.58:33916 fd:8
2019/10/25 10:36:21 [debug] 16329#16329: *452 event timer add: 8: 60000:7320992
2019/10/25 10:36:21 [debug] 16329#16329: *452 reusable connection: 1
2019/10/25 10:36:21 [debug] 16329#16329: *452 epoll add event: fd:8 op:1 ev:80002001
2019/10/25 10:36:21 [debug] 16329#16329: timer delta: 12717
2019/10/25 10:36:21 [debug] 16329#16329: worker cycle
2019/10/25 10:36:21 [debug] 16329#16329: epoll timer: 60000

grafana’s log is also not that interesting:

t=2019-10-25T10:23:17+0000 lvl=info msg="HTTP Server Listen" logger=http.server address=0.0.0.0:3000 protocol=http subUrl= socket=
t=2019-10-25T10:23:17+0000 lvl=info msg="Backend rendering via phantomJS" logger=rendering
t=2019-10-25T10:23:17+0000 lvl=warn msg="phantomJS is deprecated and will be removed in a future release. You should consider migrating from phantomJS to grafana-image-renderer plugin." logger=rendering
t=2019-10-25T10:25:25+0000 lvl=info msg="Request Completed" logger=context userId=0 orgId=0 uname= method=GET path=/admin/users/edit/2 status=302 remote_addr=127.0.0.1 time_ms=0 size=29 referer=
t=2019-10-25T10:25:30+0000 lvl=info msg="Successful Login" logger=http.server User=admin@localhost
t=2019-10-25T10:27:09+0000 lvl=info msg="Request Completed" logger=context userId=0 orgId=0 uname= method=GET path=/ status=302 remote_addr=127.0.0.1 time_ms=0 size=29 referer=
t=2019-10-25T10:30:08+0000 lvl=info msg="Successful Logout" logger=http.server User=admin@localhost
t=2019-10-25T10:30:08+0000 lvl=info msg="Request Completed" logger=context userId=1 orgId=1 uname=admin method=GET path=/logout status=302 remote_addr=127.0.0.1 time_ms=7 size=29 referer="http://grafana.amaiz.com/?orgId=1"
t=2019-10-25T10:36:08+0000 lvl=info msg="Successful Login" logger=http.server User=admin@localhost
alexey_ruzhitsky_amaiz_com@grafana-1:~$ 

Versions:
Grafana: v6.4.3 (commit: 3a2bfb7) (v 6.0.0. gives the same)
Nginx: nginx/1.14.0 ( 16.1 gives the same )

Anyone who can help me?((
Forgot to mention: current instance is on GCP.
Btw: during weekend I’ve managed to spin up local grafana instance with completely same confguration and… it was completely working one! But neither for Google Cloud!

Make sure you’re allowing PUT methods. You can see it’s submitting a PUT but you don’t see any PUT methods in your Grafana logs. So something (maybe nginx) is not allowing it.

1 Like

Thanks to everyone who tried to help me. I’ve managed to find a solution: problem resolved after enabling https! But for now I still cant understand what was wrong and why was https required? Grafana was setup to use http and I’ve tried different browsers such as FireFox, Chrome, Midori and Opera…