Hi, I’m new to Grafana (and inexperienced with databases in general) so I apologize in advance if this is a stupid/poorly-worded question. I work in a physics lab and one of the tools we use has built-in support for automatically writing experiment results to an influxdb database, but I’m having trouble manipulating the data for visualization in grafana.
I’m using an influxdb datasource and right now my query returns data of the form:
Time
dataset
pyon
t1
x
[x1 x2 …]
t1
y
[y1 y2 …]
t2
x
[x1 x2 …]
t2
y
[y1 y2 …]
…
…
…
Where x and y are string values (the names of the datasets) and xi, yi are numbers (could be integers or floats).
What I’d like to do is, for each t, select the list of values (‘pyon’) for x and y and plot them against each other (I don’t need to do a multi-series plot - one plot for each t is fine). I think I can figure out the group-by-time part of this question, although any tips are welcome, and the plotting seems fairly straightforward (using plotly), so I think my problem boils down to this: how do I extract the lists of values in the ‘pyon’ field and treat them as individual points?
Unfortunately I don’t have control over the data source, so there isn’t anything I can do on the influxdb side to structure the data in a more grafana-friendly format.
On the fact of it (I’m very happy to be shonw wrong by someone with more Influx
/ Grafana skills than I have), I think that’s simply not possible between
Grafana and Influx directly.
For one thing you do not have simple time series data (your “pyon” field is not
simple values, but instead each enttry is an array of values), and for another
thing you’re looking for an X-Y plot rather than a time-series graph (which is
what Grafana does best).
You might be able to use telegraf as a re-writer by telling it to read from
your existing Influx data source, manipulate the numbers into a Grafana-
freidnly format, and then write them to another Influx database (which could be
on the same Influx server if you want), which Grafana then gets its data from.
Given that data format, though, I don’t think it’s going to be easy without
using some full programming language such as Perl in the middle…
Yeah, I had a feeling I was trying to do something that these tools just weren’t made for.
The plotly plugin does allow for x-y plotting (i.e. not time series), and I’ve been looking at the source code to try and figure out if it would be possible to fork it and add support for parsing an array from a string and then treating that array as a series of datapoints, but it doesn’t look easy (given my lack of experience, anyway).
It’s also occurred to me that technically, the “pyon” field is also the right format to be a json array, and I see that there is some support for json datasources. Do you know if it’s possible to have one datasource feed into another? i.e. somehow pipe my influxdb datasource into a json one that could expand the array into a series of single datapoints?
It’s also possible that I should just give up and stop trying to force grafana to work in a way it isn’t meant to. After all, there are plenty of other ways to visualize data, and we will also have lots of time-series data that grafana will be useful for.
Yeah, I had a feeling I was trying to do something that these tools just
weren’t made for.
The plotly plugin does allow for x-y plotting (i.e. not time series), and
I’ve been looking at the source code to try and figure out if it would be
possible to fork it and add support for parsing an array from a string and
then treating that array as a series of datapoints, but it doesn’t look
easy (given my lack of experience, anyway).
It’s also occurred to me that technically, the “pyon” field is also the
right format to be a json array, and I see that there is some support for
json datasources. Do you know if it’s possible to have one datasource feed
into another? i.e. somehow pipe my influxdb datasource into a json one
that could expand the array into a series of single datapoints?
I think you should stop wondering how to persude Grafana to do what you want
and work more along the lines of telegraf - it’s a far more capable tool than
most people use it for, and it can certainly convert between different data
formats, as you discuss.
Whether it can take the influx JSON-like data you have and convert it to
something Grafana is happy with, I don’t know, but I’d say it’s well worth a
try.
I’ve done just what you’re looking to do by using the python influx client to query the list and then break it out into smaller messages then re-writing them back into influx. For my use case the spacing between the larger messages was broken up into smaller increments in order to display the time series correctly.
It is a bit of extra work, but I like having the ability to display spatial plots alongside my time series plots.
You can even add an orientation as a metric to display on plotly as a color gradient with a legend and all.
