Grafana 9.1.0 OSS, running AWS Linux (think RPM-based Linux).
What am I trying to do?
Provision Grafana Alert Rules and other alerting artifacts through the provisioning tools or APIs that I can script.
Why am I trying to do it?
We deploy Grafana to many different servers. To accomplish that, we deploy via “infrastructure-as-code,” and the option to manually deploy/tweak a configuration is simply not an option.
How am I trying to do it?
Here is the basic lifecycle I want to implement to deploy Grafana:
a.) define an alert rule in the UI - this increases the definition will be valid. I don’t want to have to write Alert Rules in raw JSON or YAML.
b.) export the alert rule - currently, the /api/v1/provisioning/alert-rules/ endpoint allows me to export the alert rule. The result is a JSON file representing a single alert rule that I can save as a JSON file and manage separately.
c.) provision Grafana in a template instance (an AMI in AWS-speak)- copy the JSON file into a provisioning/alerts dir. (I happen to use Ansible).
d.) deploy the instance - Incidentally, I use terraform to deploy the AMI into AWS.
e.) start Grafana at launch time - when the instance starts issuing systemctl start grafana-server.service.
f.) bonus: modify/tweak the deployed rule, then loop back to a.) Hey, why not? Grafana dashboards can easily be exported without saving to the database, so why not something similar related to alerting artifacts? (hint: I have had some success by manipulating the provenance table.)
What documentation have I used:
The documentation: Provision Grafana | Grafana documentation has the following encouraging comment:
You can manage alert objects in Grafana by adding one or more YAML or JSON configuration files in the provisioning/alerting directory.
But the example disappoints on two points:
- The format is in YAML, which is not what the APIs produce, which is JSON
- The example shows the deployment of Alert Rule Group, not an individual Alert Rule. The heading clearly says Rule, not Rule Group. Deploying several Alert Rules as a Group would complicate the administration of the rules.
When I take the JSON generated from b. and provision it, …well, crickets. I guess that’s Grafana-speak for “you’re doing something very wrong.” Hrumpf. Probably that Rule vs. Rule Group issue.
1.) Does anyone who is provisioning Grafana 9 have input?
2.) Can someone give me an example of a valid JSON Alert Rule that will deploy using the provisioning infrastructure?
3.) Can someone explain how to export a valid YAML of an Alert Rule (not Alert Rule Group) that will deploy using the provisioning folders?