Is it possible to configure alert rules for all the prometheus sources automatically?

Grafana Prometheus
1

  • What Grafana version and what operating system are you using? Ubuntu 24.04, Grafana 12.1.1

  • What are you trying to achieve? Get a simple and automatic alerting system

  • How are you trying to achieve it? I would like to setup a “alert rule” for all the prometheus sources automatically, so regardless if I add or remove a new prometheus source, the alert is set for that node

  • What happened? So there is no way to do that rn, I’ve to manually register/create the alert for each node, one by one pretty much

  • What did you expect to happen? Something easy/generic where the nodes are automatically added, the functionality exists somehow for the visualizations (you just add a variable to switch/toggle across all the prometheus sources), so I was just wondering how I could set that up for alerting.

  • Can you copy/paste the configuration(s) that you are having problems with? I don’t have any so far, I’m trying to set that up but the UI has nothing for me to configure so I was wondering how I could make that happen.

  • Did you receive any errors in the Grafana UI or in related logs? If so, please tell us exactly what they were. No errors, it’s just I don’t see an option/way to do that.

  • Did you follow any online instructions? If so, what is the URL? Not really, the YT videos are pretty simple so they don’t cover dynamic/complex scenarios like mine.

2

Hi,

do you have a separate datasource for each node? It seems rather tricky to manage, wouldn’t it be better to have agents on the nodes sending data to one prometheus instance?

If you have separate datasources for every node, I don’t think Grafana supports something like that. You have to pick a datasource for alerting. Even Infinity Plugin wouldn’t help you, as you’d need to execute multiple queries. To achieve this, you could do one of two things, one worse than the other:

  • create a datasource plugin for that (or look for one that might help you with that)
  • create a cronjob / pipeline that would create an alert for the datasource (e.g. if you’re adding nodes by Jenkins, Ansible, or something like that, call Grafana api)

If you have just one datasource and you meant instance as in instance label in the same prometheus datasource, it’s pretty doable - just create the alert and group by instance label, e.g.

min(up{}) by (instance) will return the minium value of metric up for each instance

3

Hey, first of all thank you for taking the time on go through this post.

Regarding your question: “wouldn’t it be better to have agents on the nodes sending data to one prometheus instance?” I don’t really have a problem with it as long as I am able to identify “who’s data is that”, in the other hand I liked the fact that I can just install prometheus in the server and get the instant feedback on grafana, no other service/node configuration, pretty much it works out of the box (like if you installed new relic client and you have all the data there).

And “If you have just one datasource and you meant instance as in instance label in the same prometheus datasource, it’s pretty doable - just create the alert and group by instance label, e.g.” Nope, I really meant each server has their own prometheus service, so each prometheus server is a datasource in grafana :frowning:

Interesting, I’ll keep looking into how I can point all the instances to a single prometheus to see if that works for me, thank you very much!

4

You can use Grafana Agent or Prometheus remote write protocol (each node has its own prometheus instance and they all forward the data to the “main” prometheus) to send metrics. As for “as long as I know where it’s from” - that’s what instance labels are for. You pass there node ip / node name / any other identifier of a node and let’s go.

Personally I use VictoriaMetrics. Each k8s cluster has a VMAgent and they all scrape their targets (like prometheus does) but instead of storing the data, they push to a central VictoriaMetrics Server.

2 Likes
6

Love it! I am going to try the prometheus remote write protocol, I already have prometheus in the same server as grafana so pretty much I can set it up so everyone can push data to it and grafana just read from the local prometheus instance; that sounds very doable for my setup and I am totally confortable with that, thank you so much!

7

Then if you need to check if e.g. instance is answering, you can do:

min(up{}) by (instance)

and the multidimensional alerts will add another alert instance for every instance, wether a new one or not