Exclude specific series from the alerting rules

Grafana Prometheus
1

Hello.

I’m a newbie here so I’m sorry for any incorrectness in the questions in advance :).

Trying to configure the production-only alerting system for the prometheus + grafana set up
The issue is that the the prometheus gets all metrics series via one job which is the result of the node_exporter(metrics from the DEV/QAT/PROD environments).
Is there any option to exclude DEV, QAT series from the alerting on the Grafana level(Considering the fact that all series labeled as Prod/QAT/DEV ones) ?

Thank you.

Kind regards,
Vadim

2

In our setup we also collect all metrics with just two prometheus jobs and, therefore, face similar problems. However, we have specific naming schemes for our servers, so that the name already denotes whether it is a production system or not.

Maybe you established something similar? If that’s the case you might be able to use regular expressions to filter the series by instance names. For example, all our production servers will have a p as the third letter of their name, so we can filter like this:

node_memory_MemFree{instance=~"..p.*.example.com:[0-9]+"}

HTH

3

Hello Loba,

Thank you for the quick feedback and the solution. This will work as the instances names contain the environment to which they are belong to.

Regards,
Vadim