I’m storing data from my smart home devices in VictoriaMetrics and visualize it using Grafana. This works nicely in general, though when I’m interested in very new data, Grafana leaves me hanging - it only shows me data that’s ~30 seconds old, everything newer is missing.
The problem is not divergent clocks, when I do something that causes a change in my data, I truly can see that change in my Grafana diagrams only 30 seconds later. If I manually query VictoriaMetrics using curl with one of it’s http interfaces, I don’t notice any delays, I can see the changes within the second. (but just as json / text in my terminal instead of as the beautiful Grafana diagrams I’ve grown so used to…)
I’ve got both VictoriaMetrics and Grafana running on (the same) Pi4, but looking at htop and dstat it doesn’t seem like the problem is a resource bottleneck.
Interesting enough, looking at “vmui” (the VictoriaMetrics included ui which also can generate very basic diagrams) I see the same 30 seconds delay. Only using the basic query APIs directly delivers the injected data seemingly instantaneous (with less than a second delay)
Can I somehow make Grafana show the newest data or do I need to come up with an alternative way to graph that data if I need data faster than with a 30 seconds delay?
Why do you assume that’s not a clock problem?
the explanation starts literally with the word right after what you quoted, but let’s rephrase it to make it clearer:
Because I can see that, when I perform some action that causes a clearly visible change in my data - like connecting a power device that uses lot’s of power and looking at a power meter in between - I can see that change in my Grafana diagrams only ~30 seconds later, but by directly querying VictoriaMetrics using one of it’s basic HTTP APIs, I can see the change immediately.
OK, please provide those queries. How they look like?
I’ve written down this command, but even though I thought it worked fine before, now it erroneously gives me the same value for all timestamps between now and 30 seconds ago:
curl http://192.168.16.4:8428/prometheus/api/v1/query_range -d 'query=fronius.power{device="smartMeter",type="watt",phase="sum"}' -d start=now-30s -d step=0.1s | jq -r '.data.result[] as $x | $x.values.[] as $y | [$y.[], $x.metric.phase] | @csv' | tr -d '"' | while read -r line; do IFS=',' read -rA row <<< "$line"; date -d "@${row[1]}" +"%F %T.%N = ${row[2]} (phase ${row[3]})"; done
Let me guess: it is a clock problem.
What is UTC clock difference between
- server where Grafana/Prometheus server is running
and
- machine, where browser with Grafana is running
?
why are you so fixated on it having to be a clock synchronization problem? I’m pretty sure it isn’t. Both server and clients have NTP clients setup, so there’s no chance of the clocks diverging by 30 seconds. And also I’ve told you I can see that actions I take that produce clearly visible changes in my data only are visible after around 30 seconds, exactly like grafana shows - the last 30 seconds in the graph are only straight lines.
here’s proof that the clocks are mostly in sync (didn’t hit enter at the exact same time which causes the few 100ms difference:
Because all symptoms are pointing to clock root cause. And you didn’t provide a proof until now.
Okey, so I’ll have to see if it was another API that gave me good data for the last 30 seconds, but looking at the output of this:
curl http://192.168.16.4:8428/prometheus/api/v1/query_range -d 'query=fronius.power{device="AC",type="watt"}' -d start=now-1m -d step=0.1s | jq -r '.data.result[] as $x | $x.values.[] as $y | [$y.[], $x.metric.phase] | @csv' | tr -d '"' | while read -r line; do IFS=',' read -rA row <<< "$line"; date -d "@${row[1]}" +"%F %T.%N = ${row[2]}"; done
shows me clearly the same issue as I’m seeing in Grafana, good data for everything older than the last 30 seconds, but only the same value repeated exactly for the last 30 seconds of timestamps..
VictoriaMetrics doesn’t return query results for the last 30 seconds by default, since they may be incorrectly calculated if some samples are delayed because of the interval between samples (it is usually set for up to 30 seconds in scrape configs). If you still want query results for the last 30 seconds despite the fact they may be incorrect, then pass -search.latencyOffset=1s command-line flag to VictoriaMetrics. Here is the description for this flag (it can be displayed by running VictoriaMetrics with -help flag):
The time when data points become visible in query results after the collection. It can be overridden on per-query basis via latency_offset arg. Too small value can result in incomplete last points for query results (default 30s)
1 Like
