Forwarding k8s labels to metrics in grafana-agent's Flow mode

Grafana Alloy
, ,
1

(this is a crosspost from a post from grafana-agent discussions; I’m wondering if someone here encountered a similar problem here)

Hi there,

I am migrating to the Flow mode, and I am struggling to replicate a functionality: keeping Pod labels as Prometheus metric labels.

Here is an example pod:

image
image1440×1284 92.5 KB

It exposes metrics, and those are scrapped well in my current configuration.

However, I also want my metrics to have the following labels:

app_kubernetes_io_name=ns
noselenium_tagrecords_dev_account_owner=gen_account
etc.

I have tried using the re-labeling component to no success:

// Discover Kubernetes pods to collect metrics from.
discovery.kubernetes "pods" {
  role = "pod"
}

// Scrape metrics from Kubernetes pods and send to a prometheus.remote_write
// component.
prometheus.scrape "default" {
  targets    = discovery.kubernetes.pods.targets
  forward_to = [prometheus.relabel.add_k8s_labels.receiver]
  honor_labels = true
}

prometheus.relabel "add_k8s_labels" {
  forward_to = [prometheus.remote_write.default.receiver]

  // Expose Pod labels as metric labels
  rule {
    action = "labelmap"
    regex = "__meta_kubernetes_pod_label_(.+)"
  }
}

// Collect and send metrics to a Prometheus remote_write endpoint.
prometheus.remote_write "default" {
  endpoint {
    url = "http://mimir-nginx.grafana.svc:80/api/v1/push"
  }
}

Here’s how my graph looks:

image

Here are the labels that are available on my metric:

image

Thanks for your help!

2

I think that the meta labels from the pods service discovery is being stripped before you try to save them. I would turn the “prometheus.relabel” component into a “discovery.relabel” component.

Something like this:

// Discover Kubernetes pods to collect metrics from.
discovery.kubernetes "pods" {
  role = "pod"
}

discovery.relabel "add_k8s_labels" {
  targets = discovery.kubernetes.pods.targets
  // Expose Pod labels as metric labels
  rule {
    action = "labelmap"
    regex = "__meta_kubernetes_pod_label_(.+)"
  }
}

// Scrape metrics from Kubernetes pods and send to a prometheus.remote_write
// component.
prometheus.scrape "default" {
  targets    = discovery.relabel.add_k8s_labels.targets
  forward_to = [prometheus.remote_write.default.receiver]
  honor_labels = true
}

// Collect and send metrics to a Prometheus remote_write endpoint.
prometheus.remote_write "default" {
  endpoint {
    url = "http://mimir-nginx.grafana.svc:80/api/v1/push"
  }
}
1 Like
3

@petewall hey, for some reason, I didn’t see your message.

Thank you a lot for taking the time to answer my question.

1 Like