What happened: We have a Grafana dashboard that visualizes the data in Prometheus and we have noticed that the data is not always displayed correctly. For example, the dashboard has an stat element that displays a sum of the values. When changing the time window, e.g. from 3d to 7d, no more data is found and the value 0 is displayed, although the value was greater than 0. However, if you go from 7d to 12d, the data is found again and a value greater than 0 is displayed again. However, when I query the metrics with the Grafana Explore, I find data for every time range! Sometimes the following behavior also occurs. The value 0 is displayed. After a browser reload, data is suddenly found for the period and a value greater than 0 is displayed. Additionally, I open the dashboard in different browsers and different values are displayed for the same period. I have also tried incognito/private Browsing window, without success. What could be the reason for this issue? Is there a cache? Can I configure the dashboard so that the data is always queried live?
Here is an example:
Before the browser reload
After the browser reload
What you expected to happen: The data is always displayed consistently. When I do a browser reload, the same values are displayed as before. Or if I open the dashboard in different browsers, the same data is displayed in both browsers.
Environment:
- Grafana version: 11.0.0
- Grafana runs in Kubernetes-Claster (EKS) and was installed with grafana Helm Chart without any major adjustments.
- Data source: Amazon Managed Service for Prometheus