The 7.4 release of Grafana shows impressive 30fps fucntionality. However, I seem to be unable to use this.
I have a simple InfluxDB server as a data source, freshly installed and using the newest version. Likewise, Grafana is freshly installed and using the newest version.
When I go to add a panel to my dashboard, click on a time series visualization button I can not seem to find how to add a 30 fps plot.
Can someone help me out with this? Thanks in advance!
Sorry, are we talking about 30 frames per second here?
I think I must be confused…
Antony.
Yes, 30 frames per second:
It is mentioned in this release update:
Grafana 7.4 released: Next-generation graph panel with 30 fps live streaming, Prometheus exemplar support, trace to logs, and more | Grafana Labs
And can be seen on this demo page (last plot, scroll down):
1 - New Features in v7.4 - Grafana
From what I can tell, streaming of Influx data isn’t currently supported (github issue) - partly because Influx itself doesn’t support “streaming queries” as such.
In fact, I’m not quite sure what datasources are supported in a streaming context…but it’s an interesting space to watch.
I did spend a few Google searches trying to find a stream capable time series database. But no luck.
Changing from influx to another source is within the realm of possibilities. Does anyone have recommendations in this regard, perhaps?
Empirically, according to Build a streaming data source plugin | Grafana Labs, streaming datasources should state "streaming": true in their definition.
Currently, in the master branch of Grafana, out of all native datasources in grafana/public/app/plugins/datasource at master · grafana/grafana · GitHub, only the Loki datasource has "streaming": true in its definition. Loki is probably not something you can use for time-series data in the traditional sense.
It’s possible that there are third-party plugins that do support streaming. I was recently checking out the (very nice-looking!) Redis plugin for Grafana, and it looks like it may support streaming. Indeed, the plugin.json does state "streaming": true. So maybe see if that might work? Would be interested to hear about your findings!
Thanks a lot. I love your reasoning, thanks for sharing.
Will check this out ASAP.
By the way, I ended up playing with the Redis datasource just now, and it works quite nicely as a streaming datasource for Grafana. It can read both vanilla Redis keys, as well as RedisTimeSeries key types (and a few other flavors that are probably less relevant for timeseries data).
The query UI itself was a little glitchy (i.e. I wasn’t quite sure if/when my settings were being applied while modifying the query), but once set up it seems solid.
Streaming seems to work pretty nice indeed using RedisTimeSeries with Grafana! However, just after the dashboard has loaded or when it is refreshed the plot panel starts out empty and is then filled as the data is streamed in. It would be nice if the plot is backfilled at the moment of opening the dashboard. Have you perhaps already encountered such functionality @svetb ?
@grafana89e9 yeah that’s basically how the streaming model is set up currently - the data streams into an RxJS buffer as it becomes available, and the contents of that buffer are visualized in real time.
With RedisTimeSeries, I think it’s possible to query the whole timeseries key rather than just the last value, which in theory should show you historic data also. FWIW, back when I was playing around with this, I did have a super-quick go at doing the query that way and it didn’t immediately work so I didn’t try further. Note that doing it this way might not be ideal though - since you’d be loading a bunch of data points (the historic dataset) at every refresh, rather than just the last incoming point, which is all you need after the initial load.
In principle the datasource could be amended so that it (a) queries the historic dataset on initial load, then (b) subsequently only queries the last point. Which would be ideal. But AFAIK an option for doing it that way doesn’t currently exist.
Lastly, inspired by this, I also threw together a streaming MQTT datasource: GitHub - svet-b/grafana-mqtt-datasource: MQTT streaming data source for Grafana. It’s still in “alpha” (and my first go at a Grafana plugin - or anything in TypeScript for that matter), but I feel like it turned out pretty nicely. Would be cool if it’s useful to anyone, and all feedback is welcome!
“Note that doing it this way might not be ideal though - since you’d be loading a bunch of data points (the historic dataset) at every refresh”
Just a thought: if one were to make all the plots using streams, then a refresh is only done when the user refreshes. This would solve this issue, right?
EDIT: Quick update, what I ended up doing, as a quick fix, is embedding the streaming plot as an iframe.
Steps I took:
allow_embedding = true in grafana.ini
Now the (embedded) streaming plot does not auto refresh, but all other plots in that dashboard do. Bit of a hack, but seems to work just fine.
Note that doing it this way might not be ideal though - since you’d be loading a bunch of data points (the historic dataset) at every refresh
Ah yeah I see that what I said here was ambiguous! In this context I was referring to the “live” refreshes of the panel data that are happening at 30fps; not a dashboard refresh. I called it a “refresh” because the discussion was about showing historical data in the panel as soon as the dashboard loads, and the only way I can think of for that to happen is to simply load all historical data with each request, at 30fps (which isn’t so nice…). Anyway, given that “refresh” actually means something quite different in Grafana, I think a term like “streaming panel data load” would have been more accurate 
.
Yes, that would indeed solve the issue, and I think this is the intended way for streaming visualizations to be used. Initially only partial data will be shown, while the charts are populated with new data - but after some time everything should be complete.
Yeah I hadn’t realized that data from streaming panels is cleared when the dashboard (auto-)refreshes. That’s actually pretty silly, and almost certainly maybe not the desired functionality. Indeed, it looks like in the initial implementation streaming data was not reset upon a dashboard refresh, but that behavior was subsequently changed: Don't block onMetricsPanelRefresh() when streaming data · Issue #15760 · grafana/grafana · GitHub. If you’re interested, I think it would be worth raising this as a feature request on Github.