Filter out the interpolated data points

[thainaluisa95]

I’m using Grafana with Prometheus to monitor some metrics, I’m sending metrics at irregular intervals but if this interval is big, it results in some ‘fake points’ being created, where the last sent metric is repeated over a certain period until the next metric is received. You can see this behavior in the screenshot below. I am not interested in seeing these “fake points.” Is there a way to create a query that displays only the actual values that were sent?

[grant2]

Hi @thainaluisa95

In your graph above, what do you have in the Connect null values setting, and does changing it result in the view that you are seeking?

[thainaluisa95]

hey @grant2, thanks for replying. It’s already set to ‘Never’ and changing it doesn’t impact the result. The only thing I changed in this dashboard was the Min step to 1ms, because without this some point were being ignored.

I’m using open telemetry to generate these metrics, in this case it’s a gauge. I think this behavior might come from Prometheus, because if I go to the graph in Prometheus I see the same thing.

in Prometheus:

in Grafana:

what was actually sent:

however I couldn’t find a reason for that and I was wondering if there might be a way to filter that in Grafana.

[ebabeshko]

Do you see these ‘additional’ points in Table view of your panel?

[thainaluisa95]

hey @ebabeshko, yes. I can see these points when changing to table view.

[jangaraj]

@thainaluisa95

I think this behavior might come from Prometheus, because if I go to the graph in Prometheus I see the same thing.

Correct, that’s Prometheus"feature". It’s designed for periodic metrics and sparse metrics have this problem. I think there is some prometheus server config, which may improve this behavior (try to google it), but I would recommend better metric storage, which doesn’t have this problem with sparse metrics.

Inconsistent stats from query_range with sparse metrics

[thainaluisa95]

@jangaraj I’ll look for more information about that. Thank you!

[grant2]

@jangaraj

I would recommend better metric storage, which doesn’t have this problem with sparse metrics.

Maybe InfluxDB? See this post that covers how InfluxDB handles situations where there are no values between existing values.