Grafana querying Prometheus gets data that is cut off in about a three day-old mark

  • What Grafana version and what operating system are you using?

Grafana 7.2.0 on RHEL 7.8

  • What are you trying to achieve?

Query Prometheus for certain metrics for a period of 7 days.

  • How are you trying to achieve it?

By typing in my query in the Explore window.

  • What happened?

Got the data that reaches only ~3 days back and gets cleanly cut off (no matter what metrics exactly I query). The “inspect” feature also shows raw data that is no older than three days ago, so it’s not a graph-generating related issue.

  • What did you expect to happen?

I would expect to get all the data that is stored in Prometheus (got >30 days retention, so have plenty of data to query).

  • Can you copy/paste the configuration(s) that you are having problems with?

Not sure whether relevant.

  • Did you receive any errors in the Grafana UI or in related logs? If so, please tell us exactly what they were.

No errors in the UI, no errors in the logs, even on “debug” level.

  • Did you follow any online instructions? If so, what is the URL?

No.

//

So basically for some reason my Grafana - Prometheus queries get cut off at ~3 day point (even though the data is available in Prometheus).

This does not occur when I change datasource to Cortex (that ingests the very same data that Prometheus holds).

I was able to narrow down the issue to grafana.db - when I force Grafana to use a clean database, the issue does not occur.

So it’s related to whatever configuration is stored in the database and I’d need help investigating it deeper.

Thanks!

Creating a completely new datasource that points to Prometheus does not solve this either (the problem occurs as well).