[resolved] EOF errors after upgrade

Grafana
,
1

Upgraded Grafana from version 8.5.27 to version 11.4.0 yesterday and one of the queries against an InfluxDB data source is no longer working. Nothing has changed in the configuration, but in Grafana v8 it returns the data successfully while in Grafana v11 it is getting a 500 error with the message “EOF”. Brought v8 back up and have both v8 and v11 running in parallel on different ports now and it still works fine in v8 and still fails in v11.

When I trigger this query the only thing I see in the grafana logs are the below two messages:

INFO [01-29|12:43:54] Request Completed logger=context userId=1 orgId=1 uname=admin method=GET path=/api/annotations status=200 remote_addr=[ip] time_ms=14 duration=14.192945ms size=2 referer=“http://[host]:8008/d/J7PFwj7Zk/[dashboard]?editPanel=9&from=now-1h&orgId=1&timezone=browser&to=now&var-aggregation=1&var-request=%24__all&var-testName=[testname]” handler=/api/annotations status_source=server
INFO [01-29|12:44:24] Request Completed logger=context userId=1 orgId=1 uname=admin method=POST path=/api/ds/query status=400 remote_addr=[ip] time_ms=30011 duration=30.011511531s size=64 referer=“http://[host]:8008/d/J7PFwj7Zk/[dashboard]?editPanel=9&from=now-1h&orgId=1&timezone=browser&to=now&var-aggregation=1&var-request=%24__all&var-testName=[testname]” handler=/api/ds/query status_source=downstream

In the browser’s network pane I am seeing the response of:
{“results”:{“A”:{“error”:“EOF”,“errorSource”:“”,“status”:500}}}

2

Just replying to update that I got this resolved.

The error message is incorrect – it is not an EOF, it is a timeout error. Seems like the default timeout configuration has changed, as I was able to resolve this issue by manually providing a timeout value in the datasource configuration.

2 Likes
3

Hi. I am experiencing the same errors after upgrading. What did you change the timeout value to?

4

Depends on your query, but it’s going to need LARGE values. Maybe try 300 or more, although I’ve still got some timing out with those values (even though they worked fine on the older versions of Grafana). Seems like there is some changes in the mechanism used to proxy InfluxDB requests, and the new system is EXTREMELY SLOW. Good luck. Been nearly a month and I’m still trying to fully resolve these issues. We might still need to roll back this upgrade due to this slowness.

5

starts simple - select 1 query and show it here with all query options.