How to get all or specific labels from Kube metrics into Notification Policy's Matching Label?

  • What Grafana version and what operating system are you using?
    grafana/grafana:8.2.0-beta1

  • What are you trying to achieve?
    I am trying to alert a contact based on the labels that are in kubernetes metrics. I am trying to use these labels in Notification Policy to dynamically choose the contact.
    For ex: if pod1 has a label called owner=team1 and pod2 has owner=team2, then I will have 2 contacts, 1 for each team. Then I want to create a Notification Policy to send the alert to Team1 contact if the metric has label ‘owner=team1’. Then same for Team2 if the metric as ‘owner=team2’

  • How are you trying to achieve it?
    I am able to send all pod’s labels as part of kubernetes metrics through prometheus. These labels are available in grafana. ex:
    kube_pod_labels{app_kubernetes_io_instance=“prometheus”, app_kubernetes_io_managed_by=“Helm”, app_kubernetes_io_name=“kube-state-metrics”, helm_sh_chart=“kube-state-metrics-3.4.2”, job=“kubernetes-service-endpoints”, kubernetes_name=“prometheus-kube-state-metrics”, kubernetes_namespace=“default”, kubernetes_node=“minikube”, label_app=“rkadekar”, label_pod_template_hash=“7d5655b9b4”, label_owner=“Team1”, label_test2=“test2”, label_test=“testing”, namespace=“default”, pod=“hello-world-7d5655b9b4-bmmmf”}

I am trying to use these labels in Notification Policy under matching labels section. So I can dynamically route the alerts to right contacts.

  • What happened?
    Notification policy doesn’t recognise these metric labels. It only knows Alert’s custom labels. So its not able to route it dynamically.

  • What did you expect to happen?
    I exected Notification policy to recognise these labels

  • Can you copy/paste the configuration(s) that you are having problems with?

  • Did you receive any errors in the Grafana UI or in related logs? If so, please tell us exactly what they were.

  • Did you follow any online instructions? If so, what is the URL?

1 Like