Wrong results when querying metrics sent via statsd from mimir

Grafana Prometheus
1

Our rest service generates metrics for each service call, which are send via statsd protocol for each single service call. Telegraf with StatsD Input Plugin collector acts as collector and aggregator for these. Telegraf pushes the aggregated metrics to InfluxDB using InfluxDB output and to mimir push api endpoint using http output each 15s. Both configured as recommended in influx documentation.

InfluxDB is our legacy solution, we want to switch all dashboards to mimir as metrics store.

Our target stack runs on k8s with grafana 12.2 and mimir 2.8.

This showed up a challenge probably because there is no metric, if the service is not invoked in each telegraf aggregation interval (= 15s).

We discovered, querying the same metric from mimir and influxDB shows different results. But double-checking the service calls with our accesslogs we confirmed, that InfluxDB works fine.

Compared with InfluxDB the deviation we see in mimir shows both smaller and higher values.

mimir query in grafana: (interval is set to 15s).

sum by(name) (service_requests{name="myService", host="myHost"})

influx query in grafana:

SELECT sum("value") FROM "service_requests" WHERE ("host"::tag = 'myHost' AND "name"::tag = 'myService') AND $timeFilter GROUP BY time($__interval), "name"::tag fill(null)

Results:

image
image494×240 16.1 KB

Questions:
Is anyone aware of documentation / best practises or has a comparable setup up and running?

Is it generally possible to replace influxDB with mimir in our setup, or is this impossible due to missing metrics caused by using statsd and event-based metrics push?

InfluxDB does not recommend using statsd and mimir, but for performance and scaling reasons, not functional issues.

Any help is appreciated, thanks!

2

The first thing to check is how the Telegraf StatsD counter is being represented in Mimir.

inputs.statsd has delete_counters = true by default, so StatsD counters are reset/deleted at each collection interval. If no StatsD events arrive during an interval, you should not assume that Telegraf will emit a zero-valued sample; it is important to inspect the actual samples being sent to Mimir.

I would therefore verify →

the current delete_counters setting;
the actual service_requests samples and timestamps reaching Mimir;
whether each sample represents a per-15s request count or a cumulative counter.

The last point is particularly important for the query. This:

sum by(name) (service_requests{name="myService", host="myHost"})

simply sums the sample values; it does not calculate the number of requests occurring during each time interval.

If service_requests is a cumulative counter, then a query such as:

sum by(name) (
  increase(service_requests{name="myService", host="myHost"}[$__rate_interval])
)

may be appropriate. increase() is intended for calculating the increase of a counter over a range.

If Telegraf is already sending per interval counts, however, increase() would not be appropriate.

I would compare the raw Mimir samples with the corresponding InfluxDB points first. That will help determine whether the discrepancy is occurring in the Telegraf StatsD aggregation/output stage or in the PromQL query.

Mimir provides a Prometheus compatible API and uses PromQL, so I would not conclude from the current difference that StatsD + Mimir is functionally impossible. The metric’s counter/interval semantics need to be established first.

3

Hi,

thank you very much for answering. Having a look into the values returned from mimir and influxDB was helpful, but showed up a new challenge:
These are the values from InfluxDB - which i’d also would expect to be generated from a mimir query.

image
image923×369 3.33 KB

The output from mimir shows the scale of the values is ok - but they are returned multiple times, even if no new values are pushed from telegraf.

image
image925×337 8.05 KB

So it seems, that the mimir lookback_delta (which is 300s per default) makes the difference. The value of 7 from 15:04 is filled up for the next 5 minutes in the lower graph.

Now the key question is, how do I prohibit mimir from filling up the values?
I did not find a solution.

4

Your conclusion is right. The repeated values come from Prometheus’s lookback delta, not from bad data.

An instant selector like service_requests{...} returns the newest sample within the lookback window (default 5m) at every step. Scraped series get staleness markers when they disappear, but pushed samples don’t. With delete_counters = true, Telegraf only sends a sample in intervals where events arrived, so your 7 at 15:04 is repeated until 15:09. Your Influx query uses fill(null), so it shows only the real points.

Fix: sum the samples inside each step

promql

sum by(name) (sum_over_time(service_requests{name="myService", host="myHost"}[$__interval]))

Keep the panel interval at 15s. A range selector only sees samples inside its window, so nothing is carried forward. This matches your Influx sum("value") ... GROUP BY time($__interval), since each sample is the count for one 15s interval. Steps without samples show as gaps instead of 0.

Alternative: send a cumulative counter

Set delete_counters = false in inputs.statsd, then query:

promql

sum by(name) (increase(service_requests{name="myService", host="myHost"}[$__rate_interval]))

increase() handles counter resets but extrapolates, so values can be non-integers and won’t exactly match Influx.

Try the first query and compare the hourly totals with your Influx table. If they still differ, compare the raw Mimir samples with the Influx points to see whether the gap comes from Telegraf or from the query.

5

OK - I tried summing up the samples. For almost all services (differed by the “name” tag), this works quite good, no matter which range is selected.

But for one service mimir still returns some duplicate values.
Additionally some values are not returned in the same 15s timeslot, but this shouldn’t cause any problems.

Time Mimir Mimir Time Influx Influx
10:08:00 1 10:08:00 1
10:08:15 1
10:09:00 1 10:09:00 1
10:09:15 1
10:12:45 2 10:12:30 2
10:13:00 2 10:13:00 2
10:13:15 2
10:14:30 1 10:14:15 1
10:16:00 3 10:16:00 3
10:16:15 3
10:16:30 1 10:16:15 1
10:23:00 2 10:23:00 2
10:23:15 2
10:55:30 1 10:55:15 1
11:07:00 1 11:06:45 1
11:07:15 1 11:07:00 1
11:10:30 2 11:10:15 2
11:23:00 2 11:23:00 2
11:23:15 2
12:01:45 1 12:01:30 1
12:02:00 1 12:02:00 1
12:02:15 1
12:12:00 1 12:12:00 1
12:12:15 1
Sum 36 23

Investigating this issue, I checked the telegraf agent interval and flush_interval values again - and realized, that the default value 10s is used for them. Increasing both to 15s resulted in duplicates for all services - that’s what I did not expect.

I’ll continue with my tests ..

6

I traced the data sent from telegraf to mimir remote-write endpoint and queried the results directly from mimir using the remote-read API (Grafana Mimir HTTP API | Grafana Mimir documentation).

service_requests{name="myService", host="myHost"}

Result: the metrics sent from telegraf are stored correctly in mimir and 12 requests are returned.

time count
2026-09-3015:14:50 1
2026-09-30 15:15:30 1
2026-09-30 15:16:10 1
2026-09-30 15:16:50 1
2026-09-30 15:17:30 1
2026-09-30 15:18:10 1
2026-09-30 15:18:50 1
2026-09-30 15:19:30 1
2026-09-30 15:20:10 1
2026-09-30 15:20:50 1
2026-09-30 15:21:30 1
2026-09-30 15:22:20 1

But the results of the query in grafana still differ with “Min interval” and/or “Min step” set to 15s, range is 15:14:00 - 15:23:59. 16 requests are returned.

sum by(name) (sum_over_time(service_requests{name="myService", host="myHost"}[$__interval]))
time count
2026-09-30 15:15:00 1
2026-09-30 15:15:30 1
2026-09-30 15:15:45 1
2026-09-30 15:16:15 1
2026-09-30 15:17:00 1
2026-09-30 15:17:30 1
2026-09-30 15:17:45 1
2026-09-30 15:18:15 1
2026-09-30 15:19:00 1
2026-09-30 15:19:30 1
2026-09-30 15:19:45 1
2026-09-30 15:20:15 1
2026-09-30 15:21:00 1
2026-09-30 15:21:30 1
2026-09-30 15:21:45 1
2026-09-30 15:22:30 1

I don’t have any idea, what is happening here.

But what’s very interesting:

  • Results are correct, if setting “Min interval” and/or “Min step” to 16s,17s, 24s, 26s.
  • Duplicates occur if setting “Min interval” and/or “Min step” to 18s - 23s and 25s.

I’m almost going mad about this.

7

OK - this seems to be related to grafana and / or mimir version.
All my tests, that produced strange results, were with mimir 2.8.0 and grafana 12.2.0.

Today I performed the same tests with mimir 3.1.2 and grafana 13.1.2.
Nothing strange behaviour any more, but all results are as expected.

Next step: upgrade the old setup to at least the same versions, where the results are as expected and hope, that everything will be fine.

8

I upgraded mimir from 2.8.0 to 2.17.0.
That did the job - possibly the switch from prometheus to mimir query engine (MQE) introduced with 2.17.0.

9

Glad it’s solved.

The repeated values come from Prometheus lookback. An instant selector carries the last pushed sample forward, and with delete_counters = true Telegraf only sends samples for intervals where events arrived. sum_over_time(...) over the panel interval avoids it.

The remaining odd results, which depended on Min interval/Min step, were resolved for the OP by upgrading Mimir from 2.8.0 to 2.17.0.