Grafana querying Prometheus gets data that is cut off in about a three day-old mark

Grafana
1
  • What Grafana version and what operating system are you using?

Grafana 7.2.0 on RHEL 7.8

  • What are you trying to achieve?

Query Prometheus for certain metrics for a period of 7 days.

  • How are you trying to achieve it?

By typing in my query in the Explore window.

  • What happened?

Got the data that reaches only ~3 days back and gets cleanly cut off (no matter what metrics exactly I query). The “inspect” feature also shows raw data that is no older than three days ago, so it’s not a graph-generating related issue.

image
image1773×625 56.6 KB

  • What did you expect to happen?

I would expect to get all the data that is stored in Prometheus (got >30 days retention, so have plenty of data to query).

  • Can you copy/paste the configuration(s) that you are having problems with?

Not sure whether relevant.

  • Did you receive any errors in the Grafana UI or in related logs? If so, please tell us exactly what they were.

No errors in the UI, no errors in the logs, even on “debug” level.

  • Did you follow any online instructions? If so, what is the URL?

No.

//

So basically for some reason my Grafana - Prometheus queries get cut off at ~3 day point (even though the data is available in Prometheus).

This does not occur when I change datasource to Cortex (that ingests the very same data that Prometheus holds).

I was able to narrow down the issue to grafana.db - when I force Grafana to use a clean database, the issue does not occur.

So it’s related to whatever configuration is stored in the database and I’d need help investigating it deeper.

Thanks!

2

Creating a completely new datasource that points to Prometheus does not solve this either (the problem occurs as well).

3

hi @shybbko

To troubleshoot this further, have you tried to write the same query in the prometheus UI and not through grafana?

4

Hi @mattabrams, thanks for contributing!

If I remember correctly, the issue also didn’t occur when querying the Prometheus directly. So it was related to Grafana specifically.

But some time has passed, a lot of version upgrades / service restarts etc. (changes, basically) and currently the issue is not occuring. Not sure what caused this exactly and not sure whether it will reoccur at some point. Will see, I guess :wink:

1 Like
5

Huzzah :tada:!

Well I’m pleased that this seems to have resolved itself. If it comes back, just hop back on this thread and I’ll try to get you some support from the squads

6

This topic was automatically closed after 365 days. New replies are no longer allowed.