Connection timeout error from Grafana to Postgres

Grafana PostgreSQL
1
  • What Grafana version are you using? Grafana 5.2.2
  • What database are you using? Postgresql
  • What OS are you running grafana on? Redhat 6.4
  • What did you do? trying to load Grafana home page in the browser to login to grafana
  • What was the expected result? Able to make connection and see the login page
  • What happened instead? Got 502 error, Gateway Connection timeout
  • If related to metric query / data viz: No
    users intermittently getting status code 502, Connection time out error after few minutes in the browser.
    Nginx is acting as the front end and reverse proxy.

When ever any user tries to hit the https://grafana.com . browser tries to connect to grafana home page and after few mins, user get 502 bad gateway Connection time out Error instead of getting Grafana login page.

Error log in Grafana Server:
t=2018-10-08T15:34:45-0700 lvl=eror msg=“Failed to start session” logger=context error="read tcp x.x.x.x:44282->x.x.x.x:5444: read: connection timed out”

Error in Postgres:
2018-10-09 01:23:13 EDT [22113]: [1-1] user=grafana,db=grafana,app=[unknown],client=x.x.x.xLOG: could not receive data from client: Connection timed out

2018-10-09 01:23:13 EDT [23270]: [1-1] user=grafana,db=grafana,app=[unknown],client=x.x.x.xLOG: could not receive data from client: Connection timed out

2018-10-09 02:08:17 EDT [26649]: [1-1] user=grafana,db=grafana,app=[unknown],client=x.x.x.xLOG: could not receive data from client: Connection timed out

2018-10-09 02:08:17 EDT [26719]: [1-1] user=grafana,db=grafana,app=[unknown],client=x.x.x.xLOG: could not receive data from client: Connection timed out

2018-10-09 04:15:02 EDT [23232]: [1-1] user=grafana,db=grafana,app=[unknown],client=x.x.x.xLOG: could not receive data from client: Connection timed out

2018-10-09 13:58:56 EDT [17401]: [1-1] user=grafana,db=grafana,app=[unknown],client=x.x.x.xLOG: could not receive data from client: Connection timed out

2018-10-09 13:58:56 EDT [17406]: [1-1] user=grafana,db=grafana,app=[unknown],client=x.x.x.xLOG: could not receive data from client: Connection timed out

2

I’ve been having this same exact issue for the past 2 weeks. It’s gotten worse, too. Would love to better understand what is happening. I know for a fact that the PostgreSQL database is fine because I can query it at the time that this error message gets thrown. That means it must be an issue on Grafana’s side.

3

This has continued to happen, as I’ve now set alerts around it. It occurred 8 times in the past 24 hours. Here’s a sample:

Error: tsdb.HandleRequest() response error &{read tcp x.x.x.x:y->x.x.x.x:y}: read: connection timed out

4

Still experiencing this issue. Happens roughly 5x per day. Again, I don’t believe this to be a database issue as I have other services connected and pulling data from it on a continual basis.

5

@bracco, Yes that is correct. Initially, I spent almost 3 weeks with Db assuming this is db issue. This is definitely Grafana issue as this issue got decreased when i changed the " conn_max_lifetime" to 2000 (default is 14400 seconds).
I found out this, by changing on this setting first to 8000 (2 to 3 times a day connection time out) then to 4000(1 or 2 times).
So, now i have set it as 2000.

Hope this helps you.