Hello,
We have configured a webhook endpoint to which we want to send alerts.
We noticed an issue when the alerts fire the labels and Annotations in the alert payload are merged with other alerts.
Is there some matching that we would need to do when creating notification policies to prevent the labels of two different alerts from being added to a single payload?
Grafana Version: v.9.1.4
The webhook payload that was generated by Grafana:
[
{
"Receiver": "webhook-test",
"Status": "firing",
"OrgId": "\u0001",
"Alerts": [
{
"Status": "firing",
"Labels": {
"alertname": "disk_space_test",
"app": "cert-manager",
"app_kubernetes_io_component": "controller",
"app_kubernetes_io_instance": "cert-manager",
"app_kubernetes_io_name": "cert-manager",
"app_kubernetes_io_version": "v1.9.1",
"component": "webhook-test",
"condition": "True",
"environment": "cert-manager",
"grafana_folder": "Webhook Alerting Demo",
"host": "ansible-tower.example.com",
"instance": "0.0.0.0:9402",
"job": "kubernetes-istio-pods",
"kubernetes_namespace": "cert-manager",
"kubernetes_pod_name": "cert-manager-6544c44c6b-4ngsz",
"name": "serving-cert",
"namespace": "kubeflow",
"path": "/",
"pod_template_hash": "6544c44c6b",
"team": "team1"
},
"Annotations": {
"description": "This description is from cert-manager-test",
"runbook_url": "https://google.com",
"summary": "This alert is from disk_space_test"
},
"StartsAt": "2023-03-13T12:41:50Z",
"EndsAt": "0001-01-01T00:00:00Z",
"GeneratorURL": "https://grafana-demo.example.com/alerting/grafana/gq4WNAaVz/view",
"Fingerprint": "fe07eea4cb580629",
"SilenceURL": "https://grafana-demo.example.com/alerting/silence/new?alertmanager=grafana&matcher=alertname%3Ddisk_space_test&matcher=component%3Dtest-alert&matcher=grafana_folder%3DTest+Alerting+Demo&matcher=host%3Dansible-tower.example.com&matcher=path%3D%2F&matcher=team%3Dteam1",
"DashboardURL": "",
"PanelURL": "",
"ValueString": "[ var='B' labels={host=ansible-tower.example.com, path=/, team=team1} value=68.32911448739412 ]"
}
],
"GroupLabels": {
"alertname": "disk_space_test"
},
"CommonLabels": {
"alertname": "disk_space_test",
"app": "cert-manager",
"app_kubernetes_io_component": "controller",
"app_kubernetes_io_instance": "cert-manager",
"app_kubernetes_io_name": "cert-manager",
"app_kubernetes_io_version": "v1.9.1",
"component": "webhook-test",
"condition": "True",
"environment": "cert-manager",
"grafana_folder": "Test Alerting Demo",
"host": "ansible-tower-1.example.com",
"instance": "0.0.0.0:9402",
"kubernetes_namespace": "cert-manager",
"kubernetes_pod_name": "cert-manager-6544c44c6b-4ngsz",
"name": "serving-cert",
"namespace": "kubeflow",
"path": "/",
"pod_template_hash": "6544c44c6b",
"team": "team1"
},
"CommonAnnotations": {
"description": "This description is from cert-manager-test",
"runbook_url": "https://google.com",
"summary": "this alert is from disk_space_test"
},
"ExternalURL": "https://grafana-demo.example.com/",
"Version": "1",
"GroupKey": "{}/{component=\"webhook-test\"}:{alertname=\"disk_space_test\"}",
"TruncatedAlerts": "\u0000",
"Title": "[FIRING:1] disk_space_test (test-alert test Alerting Demo ansible-tower.example.com / team1)",
"State": "alerting",
"Message": "**Firing**\n\nValue: [ var='B' labels={host=ansible, path=/, team=team1} value=68.32911448739412 ]\nLabels:\n - alertname = disk_space_test\n - component = alert\n - grafana_folder = Alerting Demo\n - host = ansible-host\n - path = /\n - team = test1\nAnnotations:\n - runbook_url = https://google.com\n - summary = This alert is from disk_space_test\nSource: https://grafana.example.com/alerting/grafana/gq4WNAaVz/view\n"
}
]
As you can see from the payload, the labels from a separate alert cert-manager are merged into the alert for disk_space_test, when only host, and path should be present in the labels.
And this is the same case for description present in Annotations as well, they are merged from the cert-manager alert.
Is the grouping of alerts that’s causing this merge of labels and Annotations?
And this is the notification policy, we are matching both the alerts by "component": "webhook-test"
If you need more information, please let us know.