Grafana Dashboards-as-Code For Newcomers

Hi – we’re trying to formalize our usage of Grafana, and one of the initial hurdles we see is how to allow dashboards to be easily edited by non-Grafana experts, and managed in source control.

I have looked through Github and google, and found that there are two approaches:

  1. Make your dashboard using the grafana UI, save the json and merge that into git.
  2. Build a dashboard generator around your specific needs.

I have found dozens of projects that went the way of #1, and no less than 6 projects that have implemented #2. I noticed that there are at least 2 presentations at upcoming Grafanacon on the topic of dashboards as code, so I know that there is a definite need for this capability. But what seems to be missing is any consolidation on this topic.

We have some basic requirements, in that it should tie into new Provisioning feature, and that it should work well with podified Grafana running on Kubernetes.

As we’re new to Grafana, we’re probably asking a question that doesn’t have an easy answer. As a new user, which one of these solutions should we adopt? We have done some basic research into each, and found that they all have positives and negatives, and they all will need some extension for our needs.

What I think we want to avoid (or at least as last resort) is building Yet Another YAML Dashboard Generator.

There are several projects for managing dashboards in a friendly format for Grafana.

GrafanaLib from WeaveWorks: https://github.com/weaveworks/grafanalib
Grafonnet-Lib from Grafana: https://github.com/grafana/grafonnet-lib
GrafYAML from Red Hat’s OpenStack team: https://docs.openstack.org/infra/grafyaml/
grafana-dash-gen from Uber: https://github.com/uber/grafana-dash-gen
Grafana Dashboards Generator https://github.com/Showmax/grafana-dashboards-generator

So my question is: do we expect any consolidation on this front? Are people interested in trying to work in that direction?

2 Likes

Not an answer but yet-another-project to manage grafana dashboards.

You can add to that list managing your grafana dashboards from SaltStack : grafana dashboards state modules (you can also manage datasources, organisations and users) The nice thing is you can generate stuff by introspecting your infrastructure and systems with saltstack. There is probably some space there for convergence or reuse of the tools you mention.

Am interested too in the answer to your questions.

Hi!

I’m the author of grafanalib, which I’ll be presenting on at GrafanaCon.

I’d be totally in favour of some sort of consolidation. I don’t have any coherent plan, but perhaps these assorted thoughts will help move things along.

grafanalib has been far more popular than I could have anticipated. We very much enjoy using it here at Weaveworks, and we’ve already merged many patches from external contributors to support cases that don’t matter very much to us here. That said, I don’t have a great deal of time to maintain it.

Had I known of grafana-dash-gen, I probably wouldn’t have written grafanalib. If grafanalib didn’t have the momentum that it does, I would seriously look into switching. GrafYAML looks interesting, and also predates grafanalib, but since its docs are fairly light-weight I’d have to do a deeper dive before I could make a call. grafonnet-lib also looks like an option.

We have found that the ability to define site-specific functions that encode local standards and best practices is key. This means using a real programming language, as I don’t think you could get this with YAML or an ordinary template system (I might be wrong though).

But once you pick a real programming language, you exclude others. Some people like Python, some people like JSonnet, others like Javascript. I doubt there’s much that would convince Javascript fans to switch to Python, or vice versa. What could we do about this?

It’s possible that we could build a machine-readable model of Grafana’s APIs and types, and use that to generate code for multiple libraries—one for each language. Ideally, this would be maintained as part of Grafana itself. I don’t know how feasible any of this is, though.

I’d note that the Kubernetes community is having similar questions and similar discussions (no link to hand, sorry). Perhaps we can learn from them?

What are the specific ways in which grafanalib doesn’t meet your needs? If you could list them here or file them as GitHub issues, that would help me better understand what’s required for consolidation, and might even get your issues fixed :wink:

Where we’d like to be is having one language for all our system config. Currently using Terraform for provisioning, YAML for Kubernetes manifests (maybe Helm soon, but Helm counts as a language for these purposes), grafanalib for dashboards, Ansible playbooks for configuring nodes, etc. I guess JSonnet & Terraform are the furthest along to being ‘one language to rule them all’.

If you’re at GrafanaCon, I would love to chat further about this.

2 Likes

Hi,

Grafonnet author here.

All the dashboard-as-code solutions are very different and it is unclear if one can win at the end. It is also unclear how we could consolidate them – the uniqueness of all the implementation is that they end up writing JSON.

I think things might change with Grafana 5 provisioning, especially because Jsonnet has a go implementation so we could imagine runtime compilation of the dashboards without jsonnet - directly by the grafana core. We are not there yet – but people who want Dashboards as code would be able to do that with nothing else than grafana itself.

2 Likes

Hi,
Thanks,
grafanalib is owsome, I have created simple graphs and they are looking nice. I want to create more complex graphs can you provide any links, info ?? by that I can explore more.

Anyone have a favorite flow for API dashboard deploy? Seems like most of the projects are leaving that as an exercise for the reader.

Grafanalib at least has a pre-packaged docker example in https://github.com/weaveworks/grafanalib/issues/57

Though I’m struggling a bit with how that’d work when you want to use the same grafana image in both a “staging” and “production” image.

Especially when the cloud resources do vary slightly between the two environments.

Will probably just start hacking something out with grafanalib and python requests, but would be nice to see an official API client or similar well-trodden path to generating and uploading dashboards.

Thanks in advance!

Hi, have any of you found a way to force the grid_position of the panels json file to be forced in the dashboard?
I generated a dashboard with 120 panels but after the first line most of the panels get messed, assuming positions and sizes of other panels.
I did try reduce to 32, 24, 16 and 8 panels, with 8 in the first row it works, but when I start adding more it gets chaotic.
It looks that this feature “The grid has a negative gravity that moves panels up if there is empty space above a panel” might be causing all these issues. I did not find a way to disable it for tests.
I will try to find the grafana version where this feature was added and downgrade to the previous one.

Hi all,

Where are we with this? … I’ve been working hard on a solution to programmatically provision Grafana dashboards. I started with grafonnet at first but it seems Grafana is being developed quicker then the lib. Not all features (parameters) were supported that I use in the dashboards.

My end result, the uploadable json, is currently created from within saltstack. A combination of saltstack’s file.managed and Jinja templating. This got me pretty far. However, I’m currently stuck on rendering the various panels and auto generate all the ID’s. I’m seeing this Maximum call stack size exceeded grafana api upload. Which is understandable, because manually editting the end result and then uploading give the desired result.

Moving panel ID’s upstream to saltstack (and it’s pillar data, being fed from a Django model) is not desirable. This give to tight coupling between systems. And I don’t really care about the ID’s as long as it works.

Would it be possible to have grafana handle ID’s of panels the same as dashboards? … Then Grafana could stay sane while rendering the dashboard. When omitting panel ID’s is see TypeError: Cannot read property 'toString' of undefined which concludes this feature is not in there yet :smiley:

Additionally my dashboard layout is fairly straight forward. See pseudo coded setup.

dashboard:
  row (title: fqdn of website):
    panel 1: responsetime
    panel 2: http status code
  row (title: next website):
    panel 1: responsetime
    panel 2: http status code
 ... etc

I love to hear tips or updates on where this is going. If you want details on how I set up the rendering of the file, feel free to send a message.

Thanks!

Regards,

Gerard.

For anybody ending up here. I’ve written an article about how I solved the upscale API based automation for our website management platform. And the integration for monitoring and alerting that comes with it.

Article here:

Code examples here:

Regards,

Gerard.