Hello Grafana Community,
We have several questions and seek your advice to ensure the acceptance of our plugin:
Actually, our use case is that we have a custom panel plugin (we don’t have any data source plugins yet). We want this panel plugin to accept two mixed, already existing data sources:
- First data source: Any data source that allows me to import XML files.
- Second data source: native InfluxDB.
Our custom panel plugin will first read XML files. Based on the information in the XML files, it will suggest some Flux queries to the user. These queries should be typed into the native InfluxDB data source query fields for the user to review and approve or change.
So, in this case, our custom panel needs access to the InfluxDB data source query fields to provide the queries. Is this possible?
If it is not possible: We will try to make one single custom data source plugin that combines both influxDB and XML importer / parser.
One of the Grafana developers told me this:
We try to avoid publishing datasources that do very similar things - so we would rather recommend contributing to the influxdb datasource.
In this case,we would be modifying the original InfluxDB data source. So a pull request should be made here?
Are there any other options to introduce our custom InfluxDB, like an XML-Influx data source that extends the existing InfluxDB data source? And in terms of code, how can our data source extend the native InfluxDB data source ? Is it possible ?
Can’t we just read the InfluxDB authentication information from our ConfigEditor, connect to InfluxDB, apply our custom logic, and query InfluxDB using the Node.js client library (Use the InfluxDB JavaScript client library | InfluxDB Cloud (TSM) Documentation)? By the way, We tested it, and it is compatible with browser js api.
Thank you !
Going to try to repeat what I think I understand here, you want to be able to suggest influx queries & possibly manipulate what the query is, on the basis of data that is read in via XML files, did I get that right?
It sounds to me like you want to manipulate queries (fundamentally) and so a panel plugin (which is typically focused on adding new kinds of visualizations) is not what you want, because at the panel level you’re usually going to be dealing with data delivered in dataframes after the query has long since happened.
It sounds like extending the InfluxDB data source is probably your best bet, to include (kinda guessing here) changing the way query binding forms work (Build a data source plugin | Grafana Plugin Tools)
Based on what you’ve said (but opinions may differ with more information) - submitting a PR to the InfluxDB data source might be a good option. @grant2 might know more or have a more nuanced take?
Sounds very custom/niche. is this for in house use or public use?
what problem are you trying to solve?
who generates these.files and what do they contain in relation to the schema of your bucket?
Thank you @davidallen5 for your reply!
Going to try to repeat what I think I understand here, you want to be able to suggest influx queries & possibly manipulate what the query is, on the basis of data that is read in via XML files, did I get that right?
Yes, exactly! You got that right!
It sounds like extending the InfluxDB data source is probably your best bet, to include (kinda guessing here) changing the way query binding forms work (Build a data source plugin | Grafana Plugin Tools).
Based on what you’ve said (but opinions may differ with more information) - submitting a PR to the InfluxDB data source might be a good option. @grant2 might know more or have a more nuanced take?
As I can see, InfluxDB is natively supported by Grafana. Making a PR to add this particular functionality for InfluxDB means it will be available for everyone who uses InfluxDB, even for people who are not interested in this functionality. Is this okay?
That’s why I thought about making a custom data source.
Is it possible to create (custom) data source that extends the native InfluxDB data source?
Do you know how I can extend the InfluxDB data source in terms of code? Is it by creating a custom data source project and then importing InfluxDB in the datasource.ts file? I tried this method, but I couldn’t import the native InfluxDB in my data source to extend it. Do you know how? Or maybe there is another way?
Thank you!
Thank you @yosiasz !
Sounds very custom/niche. is this for in house use or public use?
It’s intended for public use, as many of our partners will utilize it.
who generates these.files and what do they contain in relation to the schema of your bucket?
These XML files contain the specifications of the InfluxDB schema (including field names, etc.). We aim to develop our custom data source to read data from these files and automatically generate Flux queries based on the information extracted.
what problem are you trying to solve?
The issue I’m addressing is:
We currently have a custom panel plugin that retrieves data from an InfluxDB data source.
Here’s the current workflow:
- The user selects our custom panel plugin.
- The user chooses the InfluxDB data source.
- The user inputs their queries.
- The panel plugin displays information based on the retrieved data from InfluxDB.
The desired workflow (where the user doesn’t need to know any Flux queries):
- The user selects our custom panel plugin.
- The user chooses the InfluxDB data source.
- The user imports XML files describing the database schema.
- Flux queries are automatically generated and populated within the native InfluxDB query inputs.
- The user can either approve these queries as they are or make modifications.
@davidallen5 maybe this explains more.
1 Like
I think you heard right previously that Grafana doesn’t like to approve lots of similar data sources because it can cause confusion (which version of Influx should I use?).
As for whether the influxdb data source maintainers would welcome a PR like that – I just don’t know. But I do know how to contact them, so I’ll pass this thread along to them and see what we can find out.
1 Like
gotcha.
so xm is auto generated based on source of truth influxdb schema? why not read dynamically from source of truth then instead of this intermediary xml file
It’s not feasible because we require permission to dynamically read data from the source of truth. However, supposing that we’re able to do so, how could we automatically populate the native InfluxDB query fields with the automatically generated Flux queries? The challenge lies not in reading the XML files but rather in the subsequent step of populating the generated Flux queries into the input fields.
not feasible from your pov or from influxdb’s technical pov meaning influxdb does not allow expose schema/metadata details?
xml just imo seems so fragile. I would maybe leverage a database that your plugin leverages but I can also see the ease of use of xml which does not involve a database.
so basically your tool is a flux query helper, I think it is a cool idea but also remember flux is on the way out, being discontinued so I would hesitate to put much effort on a tool that leverages flux.
1 Like
Can it be solved by adding sensor tag with sensor name to InfluxDB metrics? I.e. so as to have the following queries:
from(bucket: "ourBucket") |> range(start: 0) |> filter(fn: (r) => r["sensor"] == "SensorA")
from(bucket: "ourBucket") |> range(start: 0) |> filter(fn: (r) => r["sensor"] == "SensorB")
And other tags for other possible mappings…
Or XML is a suggested as a workaround, because there is no access to modification of metrics?
1 Like
this below is not xml.
SensorA -> 1765448
SensorB -> 4189155
maybe it is like so
<sensors>
<sensor>
<name>SensorA</value>
<value>1765448 </value>
</sensor>
<sensor>
<name>SensorB</name>
<value>4189155</value>
</sensor>
</sensors>
dynamic flux query from buckets variable. I will let you run with the rest, possibilities are endless
1 Like
@ebabeshko
@yosiasz
Thanks a lot to both of you. Actually XML is not the real problem imo. The challenge is to populate the automatically generated Flux queries (made with the help of XML or any other thing) into the input query fields of the influxDB native Grafana data source. I don’t know if I am explaining this well…
1 Like
I think you are missing the point. this image capture is an idea that might help you to figure out approaches that you can leverage to get to where you want using dynamic usage of variables as a concept, implementation is up to you
variables using meta data of flux
the resulting flux query dynamic
from(bucket: "${buckets}") |> range(start: 0) |> filter(fn: (r) => r["_field"] == "1765448")
and this is what it looks like when run
1 Like
another way using Business Text plugin
1 Like
Hello all! Engineer from the observability squad that has the InfluxDB data source as one of our responsibilities.
I wanted to check in and offer my thoughts. This is an interesting concept for Influx!
First thought, this is working with Flux and as InfluxDB has set that in maintenance mode, so too has our squad. We are not developing features around the Flux language feature of the InfluxDB data source.
Second, if this were to be generalized to other InfluxDB languages we could not promise support in the near future for moving forward with this as our squad is in the middle of a reorg and the InfluxDB data source will be given to a new team who will require time to onboard and deal with our current issues in flight.
So, my best advice is to create an additional InfluxDB plugin, an external plugin. We recently created an external Prometheus plugin with a separate frontend package and backend library that can be used in both external Prometheus plugins and core Grafana Prometheus. We have not created this for InfluxDB and honestly it is not on our road map. My best suggestion is if you want this capability to use XML and InfluxDB with Flux you will need to do the following:
- Copy all the InfluxDB code
- Create an external data source plugin
- Paste the frontend and backend code to the external plugin
- Fix all the coupling issues where InfluxDB is importing code directly from Grafana
- Fix any additional errors, type errors, etc.
- Make the changes necessary for using XML and any additional features.
- Publish the plugin as an augmented InfluxDB plugin with this relfected in the name.
I have to say, this is not the easiest project, but the other alternative is to wait for the new squad to take ownership of the data source, create an issue in Grafana and see if it fits into their road map. Earliest I could see getting a response on this is 2-3 months.
Anyway, I want to say this is cool, thank you for your interest in InfluxDB and Grafana!
2 Likes
