We have a timer metric obtained by statsd. Let’s call it timer.X.Y.upper. The “upper” suffix indicates that it’s representing the maximum value recorded within the aggregation time window.
As an experiment I’ve graphed the value of this metric for the last 24 hours and the last 24 days. In theory the absolute maximum value in the 24-hour period should be also be visible in the 24-day period, but no, that wasn’t the case.
In short, the 24-day graph showed that the maximum value never exceeded 5 seconds, whereas there was a 7-second spike in the 24-hour graph
So in general, increasing the time window degrades the data granularity, causing confusion with rolling averages for example. I’ve tried to force graphite to return as many data points as it can to Grafana, but no dice - it generally maxes out at around 150 points for whatever reason.
Hope there is a quick solution that I’m missing for this.