How to get all or specific labels from Kube metrics into Notification Policy's Matching Label?

[rkadekar]

• What Grafana version and what operating system are you using?
  grafana/grafana:8.2.0-beta1

• What are you trying to achieve?
  I am trying to alert a contact based on the labels that are in kubernetes metrics. I am trying to use these labels in Notification Policy to dynamically choose the contact.
  For ex: if pod1 has a label called owner=team1 and pod2 has owner=team2, then I will have 2 contacts, 1 for each team. Then I want to create a Notification Policy to send the alert to Team1 contact if the metric has label ‘owner=team1’. Then same for Team2 if the metric as ‘owner=team2’

• How are you trying to achieve it?
  I am able to send all pod’s labels as part of kubernetes metrics through prometheus. These labels are available in grafana. ex:
  kube_pod_labels{app_kubernetes_io_instance=“prometheus”, app_kubernetes_io_managed_by=“Helm”, app_kubernetes_io_name=“kube-state-metrics”, helm_sh_chart=“kube-state-metrics-3.4.2”, job=“kubernetes-service-endpoints”, kubernetes_name=“prometheus-kube-state-metrics”, kubernetes_namespace=“default”, kubernetes_node=“minikube”, label_app=“rkadekar”, label_pod_template_hash=“7d5655b9b4”, label_owner=“Team1”, label_test2=“test2”, label_test=“testing”, namespace=“default”, pod=“hello-world-7d5655b9b4-bmmmf”}

I am trying to use these labels in Notification Policy under matching labels section. So I can dynamically route the alerts to right contacts.

• What happened?
  Notification policy doesn’t recognise these metric labels. It only knows Alert’s custom labels. So its not able to route it dynamically.

• What did you expect to happen?
  I exected Notification policy to recognise these labels

• Can you copy/paste the configuration(s) that you are having problems with?

• Did you receive any errors in the Grafana UI or in related logs? If so, please tell us exactly what they were.

• Did you follow any online instructions? If so, what is the URL?