Validate GitOps provisioning

Hello there,

  • What Grafana version and what operating system are you using?
    11.0 inside a k3s cluster on NixOS 23.11

  • What are you trying to achieve?
    Right now, whenever we restart the deployment to apply the new provisioning details in our repository, it can occasionally happen that Grafana will not start, because something is amiss.

Since the provisioning goes through the API, it would be great to know if there is a way to validate the provisioning first and potentially reject the changes from being applied outright and perhaps send an E-Mail.

  • How are you trying to achieve it?
    I have looked at the CLI and other resources found within the Grafana container - but I fould nothing of the likes. The API documentation also does not seem to include an endpoint specific to validation of objects and resources.

  • What happened?
    We had one “linux person” at this company prior to me joining. That other employee manually configured and bootstrapped the Grafana deployment - so, the Helm chart or Operator were not used. By now, they have left and I maintain the cluster and it’s associated applications entirely on my own - but, I do not in fact maintain the provisioned objects - like dashboards, alerts and the likes. That happens by other collegues, which do not know a whole lot about Kubernetes. Hence, the task is very two-split, where they prepare the provisioning objects and I apply them by restarting the deployment. But even so, as the former employee added the git-sync container, the newest changes are always pulled… Meaning, if something is wrong - be it the YAML formatting or such - Grafana is now offline.

  • What did you expect to happen?
    I actually expected there to be a method to verify/validate the provision objects. The path is basically a default path and some objects even get the “Provisioned” label in the UI. Hence I had thought that there was at least a tool that would let me validate files outside. My initial idea was to just launch the validation as an initContainer after the git-sync to prevent the main container from ever starting - and writing an appropriate message to /dev/termination-log

  • Can you copy/paste the configuration(s) that you are having problems with?
    It’s a private repo, apologies.

  • Did you receive any errors in the Grafana UI or in related logs? If so, please tell us exactly what they were.
    Basically, if it worked I get the UI. If it does not, I get nothing but a blank page. :slight_smile:

  • Did you follow any online instructions? If so, what is the URL?
    Honestly, no. I stumbled into this head first because once the other employee left, I was the next-most-experienced remaining. I had to learn Kubernetes, Influx, Grafana, Telegraf and many other services like that at once, almost at the same time. So no, I followed steps from the docs to understand how the provisioning works - but aside from that, not a whole lot else.

Thank you and kind regards!

do you have a pre prod environment that needs to succeed before it pushes things to prod?

“very two-split” sounds like an issue with team synergy your manager needs to address. team-sync :wink: