Issue with Athena Plugin - tmp file not found

Grafana
1

This is a bit of a dump as I can’t re-create the issue..

Date Grafana Version Athena Plugin Version Event Observations Impact
14 Oct 2025 11.6.3 3.1.6 Baseline stable period No plugin issues reported Stable
09 Mar 2026 12.3.4 3.1.8 Major upgrade window Grafana 12 + plugin upgrade No immediate issues
30 Mar 2026 12.3.4 3.1.8 First incident 2/3 nodes affected plugin.connectionUnavailable, Unix socket ENOENT
21 May 2026 12.3.4 3.1.10 Plugin upgrade No immediate regression observed Stable
28 May 2026 12.4.3 3.1.10 Grafana upgrade New Grafana version deployed Stable initially
15 Jun 2026 12.4.3 3.1.10 Second incident 1/3 nodes affected Same ENOENT socket failure, restart resolves
16 Jun 2026 (test env) 12.4.x 3.1.10 Controlled redeploy validation Multiple plugin sockets present at startup (normal behavior confirmed) No failure
Current analysis state 12.4.x 3.1.10 Ongoing investigation PID stable, plugin sockets exist at startup, failures are node-specific No further issues observed

Error example:

logger=context userId=412 orgId=61 uname=user@foo t=2026-03-30T19:47:14.594494169Z level=error msg="Request Completed" method=POST path=/api/ds/query status=500 remote_addr=127.0.0.1 time_ms=4 duration=4.016987ms size=138 referer="https://$GRAFANA_URL/d/cf6rttw6vbjswa/team-metrics-github-copilot?from=now-7d&orgId=61&timezone=browser&to=now" handler=/api/ds/query status_source=server errorReason=InternalError errorMessageID=plugin.connectionUnavailable error="rpc error: code = Unavailable desc = connection error: desc = \"transport: error while dialing: dial unix /tmp/plugin1177355429: connect: no such file or directory\""

We have three servers on live, only the server that was affected with the most recent occurrence returns files when running ls -lt /tmp/systemd-private-*/tmp/

The other two nodes return nothing…

# ls -lt /tmp/systemd-private-*/tmp/
/tmp/systemd-private-b78693c009954217bd4b76c26c3982ad-grafana-server.service-iZ0kCE/tmp/:
total 0

# ls -lt /tmp/systemd-private-*/tmp/
/tmp/systemd-private-6282c766db154bac95769500b27bc913-grafana-server.service-TanrVF/tmp/:
total 0
srwxr-x--- 1 grafana grafana 0 Jun 15 17:53 plugin3578407374
srwxr-x--- 1 grafana grafana 0 Jun 15 17:53 plugin2580163396
srwxr-x--- 1 grafana grafana 0 Jun 15 17:53 plugin2729058362
srwxr-x--- 1 grafana grafana 0 Jun 15 17:53 plugin3999404483
srwxr-x--- 1 grafana grafana 0 Jun 15 17:53 plugin2614565982
srwxr-x--- 1 grafana grafana 0 Jun 15 17:53 plugin2060277152
srwxr-x--- 1 grafana grafana 0 Jun 15 17:53 plugin2551687393

Our test env was recently redeployed and both servers contain files in /tmp. Uptime on live is currently 12 days, the service gets redeployed roughly every 28 days.

The only way to resolve the issue is to restart grafana-server..

Appreciate not much help there, but hoping someone might have come across this. We run the following plugins, only athena plugin has been affected so far..

computest-cloudwatchalarm-datasource
grafana-athena-datasource
grafana-bigquery-datasource
grafana-timestream-datasource
grafana-x-ray-datasource
yesoreyeram-infinity-datasource
grafana-pathfinder-app

Thanks!

2

Based on the error:

dial unix /tmp/plugin1177355429:
connect: no such file or directory

One possibility is that the Athena backend plugin process is no longer available and Grafana is attempting to connect to a Unix socket that no longer exists.

The next time this occurs, before restarting Grafana, could you collect:

ls -lt /tmp/systemd-private-*/tmp/

journalctl -k | grep -i "oom\|killed" | tail -20

journalctl -u grafana-server | grep -iE "plugin|exit|killed|panic|athena" | tail -50

This should help determine whether the plugin process exited unexpectedly, was terminated by the OS, or whether the socket disappeared for another reason.

3

Thanks @infofcc3, I will check that next time. I know the PID of the plugin was present throughout the lifetime of the issue up until the point I restarted the grafana-server service.