D3.js Panel Plugin - why none?

[allomorphy]

Dear all

I am having some difficulty understanding why there is no official or even community panel for the original and foundational data visualization framework, d3.js. Using d3 gives the opportunity to go close to the metal" and to build arbitrary visualisations and animation. This is a great fit with Grafana in my opinion.

There are a number of Plugins for Plotly like the currently well maintained Nline, so it is strange that no-one has developed, or rather made available to the community a d3 Plugin.

@marcusolsson has a great tutorial at tutorial

What would be nice is a React/d3 Plugin, similar to the one described in that fine tutorial, that is well maintained and plug-and-play, allowing users to edit the Panel with custom d3 code.

I wonder if there is anyone else interested in marrying Grafana with d3.js?

Thanks for Grafana and warm wishes
Eric

[jangaraj]

@allomorphy

why there is no official or even community panel for the original and foundational data visualization framework, d3.js

IMHO: it can be easy to develop something, but bigger problem will be to maintain it: resolving issues, supporting users, updating to current Grafana, used lib versions

[yosiasz]

That is a great idea. Maybe you can be that change agent and develop such plugin

Also if you look at dynamic text plugin you can now reference an external .js resource and go that route as well

Something like was done here with plotly.js

3D Scatter Plot

[mikhailvolkov]

@allomorphy D3 is excellent for custom-specific projects, and we used it multiple times to build private plugins.

D3 provides a low-level interface, while Plotly and Apache ECharts operate with higher-level data structures and are popular within the Grafana community.

[allomorphy]

Thank you @yosiasz
D3 customisability is the only way for some of our goals. Indeed, the reason for starting this thread was to gauge the appetite of the community for an open D3 panel.

If there is support for this, then I would be happy to make a real contribution to such a project.
Please indicate support in this thread or with a
(to make Grafana even greater )

Warm wishes
Eric

[yosiasz]

Get it started with a github repo and fire it up.

I have used d3 extensively in the past before I ran into grafana. D3 is very special stuff. Here is d3 using dynamic text

referencing d3 from CDN

then custom js code

[allomorphy]

Let me do that, @yosiasz!

It would be very useful if you are able to share the actual dashboards for the two great examples you gave on Mikhail’s Dynamic Text Panel. We will work on this as a prototype and for showing by example for the new project. (When I follow the fine examples above, then I cannot get the panel to reference the script for data as it keeps looking at the (required?) datasource ).

Just a thought. I am wondering if this project of an open D3 Grafana “Plug & Play” would not be a nice way to bring some of the D3 and Observable community over to Grafana.

Thank you for sharing the enthusiasm

Warm wishes
Eric

[yosiasz]

{
  "annotations": {
    "list": [
      {
        "builtIn": 1,
        "datasource": {
          "type": "grafana",
          "uid": "-- Grafana --"
        },
        "enable": true,
        "hide": true,
        "iconColor": "rgba(0, 211, 255, 1)",
        "name": "Annotations & Alerts",
        "type": "dashboard"
      }
    ]
  },
  "editable": true,
  "fiscalYearStartMonth": 0,
  "graphTooltip": 0,
  "id": 110,
  "links": [],
  "liveNow": false,
  "panels": [
    {
      "datasource": {
        "type": "mssql",
        "uid": "0saGbrE4k"
      },
      "fieldConfig": {
        "defaults": {
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              },
              {
                "color": "red",
                "value": 80
              }
            ]
          }
        },
        "overrides": []
      },
      "gridPos": {
        "h": 12,
        "w": 11,
        "x": 0,
        "y": 0
      },
      "id": 1,
      "options": {
        "content": "<!DOCTYPE html>\n<div id=\"container\"></div>",
        "defaultContent": "The query didn't return any results.",
        "editor": {
          "format": "auto",
          "height": 200,
          "language": "markdown"
        },
        "editors": [
          "default",
          "helpers"
        ],
        "everyRow": true,
        "externalScripts": [
          {
            "id": "db387191-7555-46be-ba12-675b7724c24b",
            "url": "https://cdn.jsdelivr.net/npm/d3@7.8.5/dist/d3.min.js"
          }
        ],
        "externalStyles": [],
        "helpers": "$(() => {\r\n  // Declare the chart dimensions and margins.\r\n  const width = 640;\r\n  const height = 400;\r\n  const marginTop = 20;\r\n  const marginRight = 20;\r\n  const marginBottom = 30;\r\n  const marginLeft = 40;\r\n\r\n  // Declare the x (horizontal position) scale.\r\n  const x = d3.scaleUtc()\r\n    .domain([new Date(\"2023-01-01\"), new Date(\"2024-01-01\")])\r\n    .range([marginLeft, width - marginRight]);\r\n\r\n  // Declare the y (vertical position) scale.\r\n  const y = d3.scaleLinear()\r\n    .domain([0, 100])\r\n    .range([height - marginBottom, marginTop]);\r\n\r\n  // Create the SVG container.\r\n  const svg = d3.create(\"svg\")\r\n    .attr(\"width\", width)\r\n    .attr(\"height\", height);\r\n\r\n  // Add the x-axis.\r\n  svg.append(\"g\")\r\n    .attr(\"transform\", `translate(0,${height - marginBottom})`)\r\n    .call(d3.axisBottom(x));\r\n\r\n  // Add the y-axis.\r\n  svg.append(\"g\")\r\n    .attr(\"transform\", `translate(${marginLeft},0)`)\r\n    .call(d3.axisLeft(y));\r\n\r\n  // Return the SVG element.\r\n  //console.log(svg.node())\r\n  //return svg.node();\r\n  //container.append();\r\n  document.getElementById(\"container\").appendChild(svg.node());\r\n})\r\n\r\n",
        "styles": ""
      },
      "targets": [
        {
          "datasource": {
            "type": "mssql",
            "uid": "0saGbrE4k"
          },
          "editorMode": "code",
          "format": "table",
          "rawQuery": true,
          "rawSql": "select 1 as fake",
          "refId": "A",
          "sql": {
            "columns": [
              {
                "parameters": [],
                "type": "function"
              }
            ],
            "groupBy": [
              {
                "property": {
                  "type": "string"
                },
                "type": "groupBy"
              }
            ],
            "limit": 50
          }
        }
      ],
      "title": "Panel Title",
      "type": "marcusolsson-dynamictext-panel"
    },
    {
      "datasource": {
        "type": "mssql",
        "uid": "0saGbrE4k"
      },
      "fieldConfig": {
        "defaults": {
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              },
              {
                "color": "red",
                "value": 80
              }
            ]
          }
        },
        "overrides": []
      },
      "gridPos": {
        "h": 12,
        "w": 12,
        "x": 11,
        "y": 0
      },
      "id": 2,
      "options": {
        "content": "<!DOCTYPE html>\n<div id=\"container\"></div>",
        "defaultContent": "The query didn't return any results.",
        "editor": {
          "format": "auto",
          "height": 200,
          "language": "markdown"
        },
        "editors": [
          "default",
          "helpers"
        ],
        "everyRow": true,
        "externalScripts": [
          {
            "id": "db387191-7555-46be-ba12-675b7724c24b",
            "url": "https://cdn.jsdelivr.net/npm/d3@7.8.5/dist/d3.min.js"
          }
        ],
        "externalStyles": [],
        "helpers": "$(() => {\r\n  // Declare the chart dimensions and margins.\r\n  const width = 640;\r\n  const height = 400;\r\n  const marginTop = 20;\r\n  const marginRight = 20;\r\n  const marginBottom = 30;\r\n  const marginLeft = 40;\r\n\r\n  // Declare the x (horizontal position) scale.\r\n  const x = d3.scaleUtc()\r\n    .domain([new Date(\"2023-01-01\"), new Date(\"2024-01-01\")])\r\n    .range([marginLeft, width - marginRight]);\r\n\r\n  // Declare the y (vertical position) scale.\r\n  const y = d3.scaleLinear()\r\n    .domain([0, 100])\r\n    .range([height - marginBottom, marginTop]);\r\n\r\n  // Create the SVG container.\r\n  const svg = d3.create(\"svg\")\r\n    .attr(\"width\", width)\r\n    .attr(\"height\", height);\r\n\r\n  // Add the x-axis.\r\n  svg.append(\"g\")\r\n    .attr(\"transform\", `translate(0,${height - marginBottom})`)\r\n    .call(d3.axisBottom(x));\r\n\r\n  // Add the y-axis.\r\n  svg.append(\"g\")\r\n    .attr(\"transform\", `translate(${marginLeft},0)`)\r\n    .call(d3.axisLeft(y));\r\n\r\n  // Return the SVG element.\r\n  //console.log(svg.node())\r\n  //return svg.node();\r\n  //container.append();\r\n  document.getElementById(\"container\").appendChild(svg.node());\r\n})\r\n\r\n",
        "styles": ""
      },
      "targets": [
        {
          "datasource": {
            "type": "mssql",
            "uid": "0saGbrE4k"
          },
          "editorMode": "code",
          "format": "table",
          "rawQuery": true,
          "rawSql": "select 1 as fake",
          "refId": "A",
          "sql": {
            "columns": [
              {
                "parameters": [],
                "type": "function"
              }
            ],
            "groupBy": [
              {
                "property": {
                  "type": "string"
                },
                "type": "groupBy"
              }
            ],
            "limit": 50
          }
        }
      ],
      "title": "Panel Title",
      "type": "marcusolsson-dynamictext-panel"
    }
  ],
  "refresh": "",
  "schemaVersion": 38,
  "style": "dark",
  "tags": [],
  "templating": {
    "list": []
  },
  "time": {
    "from": "now-6h",
    "to": "now"
  },
  "timepicker": {},
  "timezone": "",
  "title": "D3 Dynamic Text",
  "uid": "fbff9af5-8d71-4f4e-8430-9ecbda40e89c",
  "version": 9,
  "weekStart": ""
}

[mikhailvolkov]

@yosiasz Thank you for sharing. I will include it in the documentation.

[allomorphy]

Thank you for this.

1. Your Plotly example in dynamic test. I was failing to render it and gave up. Then, so weird, I opened my dashboard to try the d3 json that you just posted and was surprised to see the previously attempted 3D scatter Plotly working in all its glory.
2. Your d3 example was giving me the default notice of “The query did not return any data”. I had to change the DS to sqlite as I do not have mssql. This kept generating a default query, I had to replace the default query with your “SELECT 1 as fake” for it to work.

The idea is to reference these as working examples in a running instance from the Project GH. (Will need to issue a username password because we do not allow public access for GDPR reasons and I prefer not to have to maintain another instance )
Ideally we want these examples to run from a Grafana DS and not from data being called in the chart code.

What are your feelings about a Plugin that supports multiple charting frameworks, like Plotly, d3.js and Observable’s Plot?
I mean just like the dynamic text Plugin supports arbitrary js code

First attempt to replace the d3 code with a (Observable) Plot hello_world example gave a blank canvas but no error in UI or Grafana logs. (Debugging Grafana is hard).

FWIW, here is the code placed in the Javascript Code edit form :

$(() => {
  const plot = Plot.rectY({ length: 1000 }, Plot.binX({ y: "count" }, { x: Math.random })).plot();
  const div = document.querySelector("#container");
  div.append(plot);
})

[yosiasz]

@allomorphy

The idea is to reference

You lost me after the above

[allomorphy]

Does this help?
I meant so we have a Grafana intense running Plotly, D3 and Plot. Like the image below.
My idea is that this is a prototype and an illustration for what we want the new Plugin to do.

It might be useful to bring in support for example from the D3 community. I can set this up.

Regarding Plot,
I tried and failed to deploy the simple working Plot code in the dynamic text Panel. Cannot figure out why.

<!DOCTYPE html>
<div id="container"></div>
<script type="module">

import * as Plot from "https://cdn.jsdelivr.net/npm/@observablehq/plot@0.6/+esm";

const plot = Plot.rectY({length: 1000}, Plot.binX({y: "count"}, {x: Math.random})).plot();

const div = document.querySelector("#container");
div.append(plot);

</script>

If still unclear then please let me know.

[mikhailvolkov]

@allomorphy Dynamic Text panel follows the security guidelines and does not allow to import of JS Scripts and script tags inside the Content. I don’t think this plugin fits your use cases.

You are welcome to experiment with it:

1. External libraries should be imported using special options when Sanitizing is off.
2. JS Code should be a part of the JavaScript Code editor.
3. Content should contain only HTML, Markdown, and Handlebars.

Dynamic Text plugin for Grafana | Use HTML, Markdown, JavaScript and CSS | Community use cases

[allomorphy]

Thanks @mikhailvolkov
I think from what you say it was the model tag in the Plot code that is required, perhaps.

Otherwise, I followed the same procedures as for the Plotly and D3 like replace the import by entering the URL in the edit Panel Form.

As you say this Panel is not an optimal fit for these use cases and therefore our goal of getting Plot to work with the dynamic text panel is not the goal.

The goal that we were moving toward here is to make a new Plugin.
My question is whether it is feasible to have the Plugin support more than just D3 (in the way that D3, perhaps partly, does).

[yosiasz]

@allomorphy

My question is whether it is feasible to have the Plugin support more than just D3

separation of concerns. have it do only one thing and do that one thing real well.

Some monolithic do it all plugin sounds bloated and for those that want to implement only the d3 part of it will have this superfluous stuff the plugin will load imo

as great as d3 is, the one reason it is not universally adopted by your layman visualizer is because it is what I have described it in the past as being as an “artisanal visualization tool” meaning you usually write a lot of code and most of it you get from copy pasting from examples. Adoption of it is very low though it is a lot of fun. whatever d3 plugin is created should avoid this and make it super easy for a person to plot a d3 visualization.

[allomorphy]

separation of concerns. have it do only one thing and do that one thing real well.

I do not have first hand experience to author a GF Plugin so bow to this as a design specification.

as great as d3 is, the one reason it is not universally adopted by your layman visualizer is because it is what I have described it in the past as being as an “artisanal visualization tool” …

I love the word art here. Over the years many higher level charting software systems have been developed, many of these on top of d3. These are amazing and fit for 99.n% of use cases.
D3 could also be considered Data-Driven-Art
Unrestricted painting with SVG on a canvas
… for when you want to change the world with a visualisation.
Or indeed to scratch that itch which is a feature that Plotly or ECharts cannot deliver (elegantly).

Adoption of it is very low though it is a lot of fun. whatever d3 plugin is created should avoid this and make it super easy for a person to plot a d3 visualization.

Now that is a powerful idea and could be something new.
Is this not what Observable is addressing and this makes me wonder if a Plugin (perhaps sponsored by Observable) could leverage that to generate the d3.js for rendering in Grafana.

If this is not a useful idea, then I am wondering what we can do to make d3 authoring simpler.

(My first vision for this Pl;ugin was much more modest).

[yosiasz]

@allomorphy

I am wondering what we can do to make d3 authoring simpler

That would be a question for d3 author Michael

But the key word for me personally is abstraction. D3 does abstraction but not to the level grafana does it. If I want to implement time series all one needs to do in grafana is pick the time series plugin then give it some data and voila, pretty literally. That is the beauty of grafana!

With D3 you have to build and scaffold the whole thing from ground up with a lot of code.

So in order to make it easy to use d3 give users option to also do what grafana does so beautifully- plug and play quick

[allomorphy]

That would be a question for d3 author Michael

Indeed I was wondering if Torkel and Mike are in contact.
If it might be useful, I can contact both and present to them our ideas. (change agents )

So in order to make it easy to use d3 give users option to also do what grafana does so beautifully- plug and play quick

Ah!.
So we should have a D3 Panel that looks and operates like the µplot Time Series.
Spin up and connect your datasource and select a series as you say.
Looks and feels like the default Grafana experience.

Should we build a concept Specification here or start a GH Repo and we do it in the README.md?

Some high level items in the spec from my point of view:

• Clone the GF Time Series Panel for standard behaviour for example of Datasource selection. Reduce the Overrides to minimal for first version of the Plugin.

• As the project evolves, we could add more of the Overrides for increasing clickeroo configurability

• A switch in the panel to save the d3 code to an html page
  This for me is the part.
  You then take that d3 autogenerated by the Panel to your favourite Editor or IDE for ad-hoc d3 work.

• Then when done, you click another switch in the panel and the d3 is imported from html back into your GF Panel for rendering in the dashboard (and possibly for further work in GF).

• Interpolate Interpolate Interpoiate. Like GF __globals and template variables

PS. IMO, a lot of power comes from the option to use a REST endpoint datasource. This gives full control to the backend to deliver a json object to our d3 Panel. The Panel can then access multiple items for various series and other items in the received json to configure layout and annotations or anything else you need to define the d3 work.

[yosiasz]

Very ambitious but I like it. I propose the GH approach but with some much smaller plugin as POC.

time series is way too complicated and it is something that grafana itself already provides.

As a very small d3 plugin. maybe a 4x4 heat map would be cool, should be data spurxe agnostic like pretty much all GF plugins

Start small with minimal functionality and build on it. Rome was not built in one day.

A lot of the features you talk about could maybe be done in v1.0

So this 1st plugin wlll be called d3 heatmap then start a family of plugins with distinct releases.
D3 time series
D3 bars etc

Forgot about this baby I had created long time ago for guitar chord learners but put it on hold

D3 chords

GitHub - yosiasz/d3-chords

[davidharris]

For what its worth - the link to the previous tutorial will now redirect to the new developer portal site where we’ve been migrating and updating content. The d3 tutorial is actually still draft and unpublished. It has been updated to work - can view the source here - but as part of assessing which tutorials were the priority for us to maintain, this felt like it was more about using d3 than particular features of panel development, so are not currently planning on re-publishing.