How to make a durable Grafana dashboard project (versioning and testing)

Grafana Developers & API
1

Hello all,

I’m going to start a project using prometheus, opentelemetry collector and Grafana in order to monitor an app in production. I’ve already used Grafana for other smalls projects, but for this one I have other kinds of requirements.

First, I’ve found it very difficult to version grafana dashboards, ensuring that developers modifying a dashboard export it, then put it to a gitlab repo. I didn’t find any post here or tutorial explaining what is the best practice in order to collaborate while saving grafana dashboards outside grafana. Some tools are about dashboard as code, but I think I don’t understand how these tools can help me build a dashboard while versioning it.

Then, I would like to set up some kind of automated tests. Many times, I had to put some metrics in my app code, executing the code to generate the corresponding prometheus metric and at the end, watching the correct behavior in the dashboard for every use case. That’s not very durable, if the dashboard is going to live, to be modified. Let’s take an example. I have a metric to measure the number of http requests made to the url “/foo”. I have a counter for 200 responses and a counter for other http status code. My dashboard has a counter to display the percentage of failed requests. How can I test that, if I generate a bad response for my url “/foo”, my dashboard will indeed display it like I want? I’ve looked around, but I didn’t find any tool provided by Grafana or the community. I think these kinds of tests are important to ensure that you can trust the dashboard you have crafted.

Any feedback from any kind of users will be appreciated, as these questions are keys foundations of a new project.

Thank you

2

Welcome @garnet0371

Check out this folder grafana-10.0.0\conf\provisioning and this doco for versionining grafana dashboards

3

Thanks for your answer. I’ve checked a little bit but if I’ve understood correctly, I can provide from a config file the path to dashboards, so I can first clone a git repo to that path. But seems that saving them after a change is a bit complicated. And if you want to reload the dashboards, do you need to restart Grafana (e.g the pod)?

4

so that docu is saying you can provision

Config File

Data sources

Plugins

Dashboards

Alerting

etc as you can see in the folder structure

image

5

Thanks for the additional explanations. What I see is that the yaml can specify a path to import the dashboards from, but the documentation says that if I modify in Grafana these dashboards, there are not updated from the source. So when I modify a dashboard, should I export it to put it back inside the source folder?

6

@garnet0371 We explained Provisioning and how to work with provisioning files in the blog post and YouTube tutorial:

1 Like
7

Thanks @mikhailvolkov , will check it out!

For the second question, nobody is testing their dashboards here?

1 Like
8

@garnet0371 We use Cypress for E2E testing of all our community plugins and dashboards.

If you know JavaScript, I would suggest to check it out.
Grafana has a style guide and various examples for testing Grafana Core on GitHub: https://github.com/grafana/grafana/blob/main/contribute/style-guides/e2e.md

9

Thank you @mikhailvolkov, we use cypress so I will check it out too!

1 Like