I have been driven crazy by this problem today. Here it is.
First of all, the Grafana version is 5.1.2.
My prometheus data is scraped every 1 hour, which is sparse data, I think. E.g., it is collected at 9/12 13:50:15, and next will be at 9/12 14:50:15.
On Grafana with prometheus datasource, I can see the data if I select the time range as “last 6 hours”, which makes the interval on the graph, according to my screen resolution, 30s. （To my understanding, the 30s is the very interval of query that Grafana prometheus datasource sends to Prometheus server, right.)
The data shows like the following picture.
The data is discrete. Every hour, when it appears, there are 10 points spanned in 5 minutes. The 5 minutes range starts like 9/12 13:50:07, and ends like 9/12 13:55:37. (The time is approximate). That means the data can be found between xx:50 and xx:56 in every hour.
It makes sense so far. I have no question. But If I change the time range to like “last 7 days”, the data sometimes appears, sometimes doesn’t. The refresh interval is 1m.
If my local time is within xx:50-xx:56, It can show the data, otherwise no.
Actually, I understand why there is no data if my local time is not within xx:50-xx:56. Following is my understanding.
Changing to “last 7 days” makes the graph interval (i.e. the query interval) 15m. So if my local time is like 19:15 when Grafana starts to refreshing, the start time of query would be 19:15 of 7 days ago. Today is 9/12, so it would be 9/5 19:15. The subsequent queries would be 9/5 19:30, 9/5 19:45, 9/5 20:00, …, 9/12 13:15, 9/12 13:30, 9/12 13:45, 9/12 14:00, … You can see no query falls between xx:50 and xx:56. So no data can be retrieved by Grafana prometheus datasource. It misses all of the data points “just like perfect”.
I can prove my understanding. Here it is.
If I choose the time range to “9/5 00:00 to now”, and I set the time shift of the graph panel to 7m. I also be able to see the sparse data. Because this time range makes the graph(query) interval 20m, and time shift 7m makes the starting query time is 9/4 23:53 (which is fixed, not related to my local time). So the query would occur at 9/4 23:53, 9/5 00:13, …, 9/12 13:13, 9/12 13:33, 9/12 13:53, …, 9/12 14:53, …, 9/12 15:53. As you can see, there are queries occurring at xx:53 in every hour, which is between xx:50 and xx:56. That’s why I am able to see the sparse data under this settings.
So now, If I understand it right, my question is, how can I get the data shown whatever the time range I select, or the question is like, how can I make Grafana query data at some certain time no matter what the query interval and query starting time is, like between xx:50 and xx:56.
Thank you very very much in advance!