How to manage overlaping labels in metrics and alerts

Grafana Alerting
1

If I’m querying a metric with a label component, I cannot use that label in my alert labels, even if they have different meanings.

Grafana emits a warning: level=warn msg="Evaluation result contains either reserved labels or labels declared in the rules. Those labels from the result will be ignored" labels="component=broker"

I don’t want to create a different label name for the alerts just to avoid clashing with a random metric we happen to have.

In the example above, component means the internal component of that specific software. While at the alert level, component means that specific software as part of a much bigger stack.

Any advice on how to handle this?

2

Generally speaking the best way to avoid this situation is to plan what labels you want to reserve and what labels you want to get from the query when planning out your rules and notification policies. Duplicate labels are not allowed, and if they occur Grafana will reject the duplicate label from the query instead of the label defined in the rule. I’m afraid in either case you’ll need to change the duplicate label in your rules, or reserve it in such a way that no queries can return data with that label.