Bizarre disappearing metrics (Alloy + Mimir + Grafana)

Grafana Prometheus
, ,
1

I have latest Alloy + Mimir Distributed + Grafana + Loki (the latter isn’t relevant here).

Alloy scrapes metrics and pushes them into Mimir Prometheus endpoint.
Metrics get processed errors and warnings are few, nothing remarkable.

However, there is something extremely bizarre going with Mimir querier. For last 6 hrs I get full graphs for metrics series. For last 3hrs I get barely any data.

Last 6hrs:

Screenshot_20251022_210500
Screenshot_20251022_2105001920×1144 189 KB

Last 3hrs:

Screenshot_20251022_210520
Screenshot_20251022_2105202180×1299 310 KB

You can clearly see that the data for the last 3hrs is as available as for the last 6hrs.

Any ideas what the issue may be?

2

I using custom ranges now-5h and now-4h scales. Data disappears at now-3h.

3

Checking further at now-188m data is there. At now-187m the data disappears.

4

The problem is that the default scrape was “60s”/“1m”. Changing to “5s” for default scrape interval and “4s” timeout resolved the problem.