Hello, I’m working on a project for my university with the goal to monitor elder people and detect if they have Covid-19 by tracking their heartrate, temperature, oxygen saturation, etc. I’ve written a program in Python that uploads the current value of these variables every 10 seconds to a Influx DB database and then use Grafana to show visualize how these variables change over time. This works great. To calculate these values ECG- and PPG-signals were used and I would also also like to visualize these signals but the problem is that these signals are typically sampled at a high frequency like 100 Hz and it’s impossible for me to upload a sample 100 times a second to the database. So I was wondering if its possible for me to upload for example a list of 100 (or even 1000 samples) to the database and then let Grafana plot this list.
Maybe there is another solution? The goal is just to visualize the ECG and PPG signal.
Thank you very much in advance!
What you’re describing should work. Influx supports nanosecond-precision timestamps, and Grafana should be able to visualize data with that kind of resolution. Are you facing a specific issue getting this to work?
Since we are trying to send 100 values every second, we put them in JSON arrays.
{"ID":"Unit_A", "Waveform1":[1,2,3....,100], "Waveform2":[1,2,3,...,100], ..... , "Waveform10":[1,2,3,...,100]}
When I used Telegraf to pipeline the MQTT to influxdb, it made many fields like Waveform1_0, Waveform1_99, etc. What I wanted was to have one field, and then create new timestamps at a constant rate. Is there a way for telegraf or some other tool to do that? I have complete control over the JSON format, but if I have to add names and timestamps for each value to the JSON, the wireless link will be overloaded.
Also, as a second issue, I am reading the Grafana will probably struggle trying to run the queries even several times a second to get a real time display of thousands of points, because it’s having to query them all, rather than just the last packet.
Thanks @mikemitchell for the extra info, this is very helpful!
Re. getting the data into Influx, I have to admit my knowledge of Telegraf is limited. Since it belongs to the Influx stack, it may be better to get some advice on the Influx forum. There’s a “Collect” category where I’ve seen similar topics being discussed.
Of course if you happened to have a Python script or similar that’s used for data ingestion (like OP @msaenen77), then adding timestamps through that would be pretty straightforward.
Got you - I didn’t fully realize that real-time visualization is the goal here (it didn’t seem to be in OP’s post). This does indeed change the picture a little, though I feel like it should still be workable through Grafana. Maybe consider the following two points:
- Grafana’s support for streaming data has been steadily improving, and I believe is currently quite mature. Check out How to Stream Sensor Data with Grafana and InfluxDB | Grafana Labs and Grafana 7.4 released: Next-generation graph panel with 30 fps live streaming, Prometheus exemplar support, trace to logs, and more | Grafana Labs
- Even without “true” live streaming, you can in practice set the Grafana refresh rate to something super-quick like 0.1s or 0.2s (5-10Hz), which is (a) fast enough to appear as a smooth flow to a human observer, and (b) in many cases something that your server/network/browser can cope with - even if you’re loading the whole chart (~1,000 data points) with each request. YMMV wildly, of course, but might be worth a shot. See Lowest Refresh Rate Now 5s? and Configure Grafana | Grafana documentation about enabling fast refresh rates.
Good luck! Would love to hear how you get on in the end.
@svetb Thanks, that is quite helpful. I was probably reading some older posts, and did not realize that fast streaming is locked out in the default config file min_refersh_interval . I will try that.
Hi msaenen77, how did your project go?
I have a similar project, could you help me?