After Upgrade from Grafana 5 to 6, dashboard keeps logging out

#1

Hi, we upgraded to Grafana 6, now our TV dashboard logs out itself every few hours.

Did something change in Grafana 6 that’s shortened the session time?

I added GF_SESSION_LIFE_TIME: 604800 to our env vars (since we are using Docker), but made no difference.

Any advice appreciated.

Thanks

Automatic log out Grafana session
#2

No advice other than same things happens to me now in v6.
Guess it has something to do with http://docs.grafana.org/guides/whats-new-in-v6-0/#easier-to-deploy-improved-security

#3

Might be related indeed. I can say, in the mean time we have downgraded to v5, since we are running in kiosk mode on a chrome bit, and it’s a pain to login.

#4

Same issue here!! Now we have public dashboards at login screens several times a day. Please help!

Thank you!

#5

Joining the bandwagon. Kiosk mode is supposed to be login-once-and-forever, there is no documentation on how to set up proper kiosk mode. We’re still on 5.4.3 since I was afraid the above issue will arise.

Can somebody try to “share” the dashboard via URL and use that URL in kiosk mode?

EDIT. I tried this in Grafana 5.4.3 via “share snapshot” - it only displays currently displayed time range and no alerts from datasources like Zabbix. Sharing by link requires users to log in.

#6

Are you using a local grafana account for the kiosk user or LDAP integration? I haven’t had any logouts with local accounts with v6.

#7

We are using local accounts and I created a new account today just for the kiosks to see if a “fresh 6.x made account” made a difference. No such luck. Still logging itself out.

#8

After some testing, this appears to be specific to dashboards as I left one of our machines on the “choose dashboard” screen and it did not log out after 24+ hours. If I choose a dashboard - any dashboard - it logs out after a period of time (I am trying to capture the exact time cycle). It does the same when a playlist is running.

#9

one addition step - if you can leave it logged in on a dashboard, and keep chrome devtools open to the console window with “preserve log” enabled, we might be able see what is triggering the logout.

a simple dashboard (a graph with no metrics) can be tried too, in case it is a datasource that is triggering the logout.

#10

Sorry - it’s been a while and I have been out of the office.

The test with a graph with no datasource seems to do the trick. Grafana did NOT log out on it’s own in a 72+ hour period. Trying to watch what happens with Chrome / inspector just results in what appears to be a normal log out and is really hard to catch. Our datasource is Zabbix btw. Thank!

#11

Update: I was able to keep a dashboard open / up for 48 hours WITHOUT a refresh interval. Once I added the interval, it logged me out in 30-40 mins.

#12

Has anyone found a fix for this yet? My dashboards are still down… Thanks!

#13

Hello!
We are having the same problem! We can no longer leave the dashboards on the TVS. Is this a bug on this version?
Thanks!

#14

Can you not scroll back in the window to see in detail what happened?
[Edit] In fact I believe you can save the log to file for easier analysis.
https://support.shortpoint.com/support/solutions/articles/1000222881-save-google-chrome-browser-s-console-file

#15

I can see 2 GET, and in each response it tries to “Set-Cookie” for the session (different values)
GET /api/alerts/states-for-dashboard?dashboardId=2

Set-Cookie: grafana_session=b64e73f4cef599d20dca5152d03fa6e9; Path=/; Max-Age=2595600; HttpOnly; SameSite=Lax

GET /api/datasources/proxy/2/query?....
Set-Cookie: grafana_session=bf6f29d7d1d4eb73afd16de567859300; Path=/; Max-Age=2595600; HttpOnly; SameSite=Lax

In the third and fourth request I get 401 response back with
Set-Cookie: grafana_session=; Path=/; Max-Age=0; HttpOnly; SameSite=Lax

#16

I have been trying to capture this for a few days. In my case, and that of a co-worker, if inspector is open the tab crashes and we lose everything. If inspector is not open, the session logs out as described above.

#17

A couple of ideas, since the user session has changed in v6, there are other parameters.

The default is to refresh the auth token every 10 minutes, either the kiosk is having trouble connecting to grafana periodically (server load, network traffic, anything that would interrupt), or the refresh is somehow broken in the code base.

Try extending the refresh interval and see if it prevents the logout (for 10 hours)

in docker:

GF_AUTH_TOKEN_ROTATION_INTERVAL_MINUTES=600

or in /etc/grafana/grafana.ini

[auth]
token_rotation_interval_minutes=600

It would also be interesting to see if your grafana logfile has anything about the logout.
If you enable debug level logging, the auth/token code will give a fair amount of output that can be useful.

#18

Thank you! I have just made the change to the ini file (which feels weird on *nix :wink: and will wait to see how it goes. Thanks!

#19

This change did not remediate the problem. Dashboards are still logging out. Working on the log capture now.

#20

So it logged out and crashed the browser over night. Looking for “eror” in the logs, I have this:

t=2019-04-22T10:25:26-0400 lvl= eror msg=“failed to look up user based on cookie” logger=context error=“user token not found”

t=2019-04-22T13:11:56-0400 lvl= eror msg=“Server shutdown” logger=server reason=“System signal: terminated”

t=2019-04-22T13:12:00-0400 lvl= eror msg=“can’t read datasource provisioning files from directory” logger=provisioning.datasources path=/etc/grafana/provisioning/datasources error=“open /etc/grafana/provisioning/datasources: no such file or directory”

t=2019-04-22T13:12:00-0400 lvl= eror msg=“Can’t read alert notification provisioning files from directory” logger=provisioning.notifiers path=/etc/grafana/provisioning/notifiers error=“open /etc/grafana/provisioning/notifiers: no such file or directory”

t=2019-04-22T13:12:00-0400 lvl= eror msg=“can’t read dashboard provisioning files from directory” logger=provisioning.dashboard path=/etc/grafana/provisioning/dashboards error=“open /etc/grafana/provisioning/dashboards: no such file or directory”


NOTE: I have also removed all cookies and browser data prior to running these tests.

Same log with 5 lines before and after each error:

0-t=2019-04-22T07:31:12-0400 lvl=info msg=“cleanup of expired auth tokens done” logger=auth count=0

98:t=2019-04-22T10:25:26-0400 lvl= eror msg=“failed to look up user based on cookie” logger=context error=“user token not found”

223-t=2019-04-22T13:04:57-0400 lvl=info msg=“Request Completed” logger=context userId=2 orgId=1 uname=cbateson method=GET path=/logout status=302 remote_addr=172.21.5.138 time_ms=73 size=29 referer=“http://zabbix:3000/d/wkCBX0jmz/dirac-cluster?orgId=1

472-t=2019-04-22T13:11:56-0400 lvl=info msg=“Shutdown started” logger=server reason=“System signal: terminated”

580-t=2019-04-22T13:11:56-0400 lvl=info msg=“Stopped Stream Manager”

645-t=2019-04-22T13:11:56-0400 lvl=info msg=“Stopped TracingService” logger=server reason=nil

735-t=2019-04-22T13:11:56-0400 lvl=info msg=“Stopped CleanUpService” logger=server reason=“context canceled”

1255-t=2019-04-22T13:11:56-0400 lvl=info msg=“Stopped UserAuthTokenService” logger=server reason=“context canceled”

1366-t=2019-04-22T13:11:56-0400 lvl=info msg=“Stopped UsageStatsService” logger=server reason=“context canceled”

1474-t=2019-04-22T13:11:56-0400 lvl=info msg=“Stopped ProvisioningService” logger=server reason=“context canceled”

1584-t=2019-04-22T13:11:56-0400 lvl=info msg=“Stopped NotificationService” logger=server reason=“context canceled”

1694-t=2019-04-22T13:11:56-0400 lvl=info msg=“Stopped HTTPServer” logger=server reason=nil

1780:t=2019-04-22T13:11:56-0400 lvl= eror msg=“Server shutdown” logger=server reason=“System signal: terminated”

1887-t=2019-04-22T13:11:59-0400 lvl=info msg=“Starting Grafana” logger=server version=6.0.1 commit=0c44a04 branch=HEAD compiled=2019-03-06T09:21:49-0500

2035-t=2019-04-22T13:11:59-0400 lvl=info msg=“Config loaded from” logger=settings file=/usr/share/grafana/conf/defaults.ini

2154-t=2019-04-22T13:11:59-0400 lvl=info msg=“Config loaded from” logger=settings file=/etc/grafana/grafana.ini

2261-t=2019-04-22T13:11:59-0400 lvl=info msg=“Config overridden from command line” logger=settings arg=“default.paths.data=/var/lib/grafana”

2397-t=2019-04-22T13:11:59-0400 lvl=info msg=“Config overridden from command line” logger=settings arg=“default.paths.logs=/var/log/grafana”

5418-t=2019-04-22T13:12:00-0400 lvl=info msg=“Initializing UsageStatsService” logger=server

5505-t=2019-04-22T13:12:00-0400 lvl=info msg=“Initializing UserAuthTokenService” logger=server

5595-t=2019-04-22T13:12:00-0400 lvl=info msg=“Initializing CleanUpService” logger=server

5679-t=2019-04-22T13:12:00-0400 lvl=info msg=“Initializing NotificationService” logger=server

5768-t=2019-04-22T13:12:00-0400 lvl=info msg=“Initializing ProvisioningService” logger=server

5857:t=2019-04-22T13:12:00-0400 lvl= eror msg=“can’t read datasource provisioning files from directory” logger=provisioning.datasources path=/etc/grafana/provisioning/datasources error=“open /etc/grafana/provisioning/datasources: no such file or directory”

6108:t=2019-04-22T13:12:00-0400 lvl= eror msg=“Can’t read alert notification provisioning files from directory” logger=provisioning.notifiers path=/etc/grafana/provisioning/notifiers error=“open /etc/grafana/provisioning/notifiers: no such file or directory”

6361-t=2019-04-22T13:12:00-0400 lvl=info msg=“Initializing TracingService” logger=server

6445:t=2019-04-22T13:12:00-0400 lvl= eror msg=“can’t read dashboard provisioning files from directory” logger=provisioning.dashboard path=/etc/grafana/provisioning/dashboards error=“open /etc/grafana/provisioning/dashboards: no such file or directory”

6691-t=2019-04-22T13:12:00-0400 lvl=info msg=“Initializing Stream Manager”

6761-t=2019-04-22T13:12:00-0400 lvl=info msg=“HTTP Server Listen” logger=http.server address=0.0.0.0:3000 protocol=http subUrl= socket=

6892-t=2019-04-22T20:12:00-0400 lvl=info msg=“cleanup of expired auth tokens done” logger=auth count=2