Hi,
I creted Alloy pipeline whcih works fine, and send logs to Loki, now I am tring to configure alerting with Alert Rules in Grafana Loki. How to remove all labels from alert in Alert Manager which are send from Loki starting from __, like __alert_rule_uid__, __grafana_autogenerated_, __grafana_receiver__ and __grafana_route_settings_hash__?
I was trying to aggregate them on Grafana Loki, but it’s still showing. I found that article, however I can’t find those options directly in my view: Reducing alert labels There it was using Prometheus as the source, but I am using Loki.
You can remove unwanted labels in your metrics query with a sum by (<LABELS>)
It doesn’t work when I set query metric like that:
sum by (job) (
count_over_time({job=“coin”}
)
Could labels be coming from outside of Loki? What are you using to run the rules? Ruler?
I don’t understand or I’m not familiar with the Ruler tool at all. I create alerts directly from Grafana using Alert rules. During configuration, I can group labels or add new labels, but I don’t see any way to remove all the labels I mentioned above. They’re always automatically added to Alert Manager by default.
Then it’s likely that those labels are generated by Grafana. I don’t use Grafana alerts myself, so I can’t really comment on whether it can be removed.
1 Like
Hi @kreg870,
You’re absolutely right — Grafana automatically adds some internal labels when sending alerts.
Just to clarify a bit so I can give you the most helpful advice:
Labels like __alert_rule_uid__ and other internal metadata are primarily for Grafana’s internal use — for example, to link back to the alert rule in the UI. These usually don’t appear in end-user notifications .
Could you share a bit more about why you want to remove them? That’ll help determine whether this can be handled via the notification template, or if another approach is more suitable.
Thank you @pepecano for you reply.
I’m using an external Alertmanager. The labels like __alert_rule_uid__, __grafana_autogenerated__, __grafana_receiver__, and __grafana_route_settings_hash__ are visible only in the payload, as shown in the screenshot. There are more labels which I removed for that case.
I’d like to remove these autogenerated labels to keep the alert clean and focused on the labels that are important during an incident.
We maintain a few key labels (like environment, alert name, and alert ID) which are essential for us, and we’d prefer to avoid clutter from these additional system labels.
Hi again @kreg870,
In Grafana, you can disable some auto-generated labels using the disabled_labels config option. However, it’s unclear whether this affects internal labels like __alert_rule_uid__ or __grafana_route_settings_hash__, as the option currently applies only to reserved Grafana labels like grafana_folder.
If you’re forwarding to receivers like email, Slack, webhooks, etc., you can customize templates to hide labels you don’t want to display — though each receiver supports only a limited set of templated fields.
For additional context, the Grafana built-in Alertmanager recently added support for customizing the full webhook payload (PR #103818), which gives full control over the alert payload. However, this isn’t currently possible when using an external Alertmanager.
Since Prometheus Alertmanager doesn’t support label rewriting or filtering out unwanted labels, a common solution is to route alerts through a webhook proxy that can sanitize or transform the payload before forwarding it to your actual destination.
Hope it helps!
Thanks @pepecano for your support.
Yes, you’re correct. Even when I set the labels with __ in the Grafana configuration file and restart Grafana, then trigger a specific alert to Alert Manager, I still see all the labels with __ unfortunately. It works as you suggested most likely, i.e., with labels like grafana_folder, but not for those which starts from __.
