Rate()[1m] missing value mystery

Hi all,

I have rate()[1m] not return data issue at some datapoints. And I am trying to understand it.

In my system, the scrape_interval is 15s and my query looks like: rate(grpc_server_requests_seconds_count{namespace=~"$namespace", release=~"$release", app=“nlu-orchestrator”, statusCode=“OK”}[1m]), a few data will be missing as shown below. if I remove rate()[1m] and rerun the query, I could get all datapoints with grpc_server_requests_seconds_count.

I put raw data side by side for the same release, same container as below. As you can see, at the time at 2:56:30, the one with rate()[1m] don’t have value continuously for 4 times. However, on the left side without rate(), there are values. These 4 values in green should be used for calculate the 1st red entry at 2:56:30 with rate()[1m], but we got an empty cell.

The rate() calculation is still a mystery for me. If anyone could give me an example how it was calculated exactly, that would be really helpful! Thanks a lot!

It is likely the values are missing on the time range where rate() returns nothing. See the following article on how to fetch raw samples from the database - Analyzing Prometheus data with external tools | by Aliaksandr Valialkin | Medium

Also very useful is this awesome breakdown from one of the Prometheus maintainers