Prometheus DS "metric request error" new error on upgrading ubuntu

I’ve had an instance of Graphana running for 2 years on an Ubuntu box using Prometheus as a data source.

After updating ubuntu via apt I noticed my all my dashboards went down.
Going to the Prometheus DataSource settings I see the error
Error reading Prometheus: Metric request error

Looking in my apt upgrade log it looks like things broke with the update batch

Upgrade: libglapi-mesa:amd64 (21.0.3-0ubuntu0.3~20.04.4, 21.0.3-0ubuntu0.3~20.04.5),
libgl1-mesa-dri:amd64 (21.0.3-0ubuntu0.3~20.04.4, 21.0.3-0ubuntu0.3~20.04.5), 
grafana:amd64 (8.2.5, 8.3.0), 
mesa-vulkan-drivers:amd64 (21.0.3-0ubuntu0.3~20.04.4, 21.0.3-0ubuntu0.3~20.04.5), 
libglx-mesa0:amd64 (21.0.3-0ubuntu0.3~20.04.4, 21.0.3-0ubuntu0.3~20.04.5)

If I use curl to access Prometheus from the same machine I get results for endpoints like /metrics or api/v1/query. So it appears that Prometheus is working fine.

Also, googling the error message Error reading Prometheus: Metric request error returns zero results. So, not sure what to do here … roll back the grafana version or contact the Ubuntu apt maintainers?

Post this in the ubuntu stack exchange as well (in case there’s an issue with their grafana package)

Downgrading from grafana/stable 8.3.0 to grafana/stable 8.2.5 resolves the issue (until I need to upgrade grafana).

This is using the archive https://packages.grafana.com/oss/deb

Tried the ubuntu grafana update grafana/stable 8.3.2 amd64 and it still breaks the Prometheus DataSource.

Hello, I had the same issue upgrading grafana from 8.0.4 to 8.3.1:
Error reading Prometheus: Metric request

On some dashboard I had a ‘min step’ defined, removing it (to let it default to auto) solve the issue on my side.

Edit: it seems we used a wrong syntax for “Min step”: 5min ==> 5m solved it

1 Like

Same here. Were using an interval variable and something didn’t like the value. Cleared it out and all worked well.