Max_over_time not showing actual maximum

Grafana Prometheus
1

I’m trying to get the max container memory usage as a percentage value over a time range, and I’m quite close, but my current query seems to fall short.

As can be seen in the image below I’m getting two time series for memory usage by ntfy, and a single max value as a result. The problem is that the max value shown in the table is not the actual maximum found in the time range, the highest value is 13%, which is in the series with the yellow line. I assume this has something to do with the fact that the yellow series ends and is replaced by the green one, i.e. the ntfy container was replaced. But why is the query result only considering values from the green series and not values across the entire time range?

The query in code for reference:

sum(100*(max_over_time(container_memory_max_usage_bytes{container_label_com_hashicorp_nomad_task_name="ntfy"}[$__interval])/container_spec_memory_limit_bytes{container_label_com_hashicorp_nomad_task_name="ntfy"})<100) by(container_label_com_hashicorp_nomad_alloc_id,container_label_com_hashicorp_nomad_job_id,container_label_com_hashicorp_nomad_task_name)

image
image2037×1056 166 KB

2

Hi,
First of all, max_over_time(selector[$__interval]) will use $__interval as the time it should be looking back for max value - for most cases this value is small - I think it equals how far the points are afar, so you won’t get far with that. I think you’d get more value out of using $__range if you want to get max value in selected range.

Try that and see if it resolves your issue. There still might be something with the end of the series, but I’d need to know more about what you want to get out of this query (what do you want your query to do?).

3

That’s a good suggestion, I wont be able to try it for a while though as I’m out traveling for a bit.

What I want the query to do is to give me the max value of any series returned, ie the highest value of any container grouped by the labels. The goal is for me to be able to scale the memory allocations of my containers better when I can see what their memory usage peaks at.