CoreDNS integration not working and integration instructions confusing

Grafana Cloud
1
  • What Grafana version and what operating system are you using?

Grafana Cloud

  • What are you trying to achieve?

Trying to add the CoreDNS integration to the Kubernetes integration

  • How are you trying to achieve it?

Following the instructions I created an extraConfig file amended it to match my labels and port:

discovery.relabel "coredns" {
  targets = discovery.kubernetes.pods.targets
  rule {
    source_labels = ["__meta_kubernetes_pod_label_k8s-app"]
    regex = "kube-dns"
    action = "keep"
  }
  rule {
    source_labels = ["__meta_kubernetes_pod_container_port_number"]
    regex = "9153"
    action = "keep"
  }  
  rule {
    source_labels = ["__meta_kubernetes_pod_name"]
    target_label = "instance"
  }
}

prometheus.scrape "coredns" {
  job_name     = "integrations/coredns"
  targets      = discovery.relabel.coredns.output
  honor_labels = true
  forward_to   = [prometheus.relabel.metrics_service.receiver]
}

and applied it re-applying the the helm upgrade command adding the argument --set-file extraConfig=values.alloy and tapped the Test Integration once the Helm command had finished.

  • What happened?

I get the error " No metrics were found for this integration.". There are two other tabs on the integration page one called “Agent” and one called “Agent Operator” it’s not clear if I ave to do anything with these or whether they are alternative instructions, the UI is very unclear about this. Also given the helm installed doesn’t even install the agent as it’s deprecated it installs Alloy.

I’ve verified on my cluster that core-dns has the cache and Prometheus plugins, I’ve tested the /metrics end-point by port-forward to my machine and testing curl and can see events being emitted by CoreDNS. Every on my cluster end seems properly configured and I’ve validated the labels and port matches my configuration.

  • What did you expect to happen?

For the connection test to pass.

  • Did you receive any errors in the Grafana UI or in related logs? If so, please tell us exactly what they were.

“No metrics were found for this integration.”

  • Did you follow any online instructions? If so, what is the URL?

Grafana though the CoreDNS integration popup doesn’t have a URL so I can’t paste a link to it. Instead screenshot below of what it looks like.

Screenshot 2024-04-22 at 16.51.08
Screenshot 2024-04-22 at 16.51.081592×1254 172 KB

  • More information

Given I’ve ruled out everything on my end I’m at a loss on how to investigate further, there doesn’t seem to be a way of accessing the cloud-hosted Prometheus instance and viewing the targets or configuration.

It’s worth noting my Kubernetes instance is partially managed by Digital Ocean.

I’m on the free trial and I have to say the process so far has been far from the smooth sailing advertised.

Any help would be greatly apreciated.

2

Try changing the __meta_kubernetes_pod_label_k8s-app label to __meta_kubernetes_pod_label_k8s_app in your discovery.relabel component.

In metrics-world, all special characters become underscores, so your pod with the k8s-app label will be set to the metric meta-label __meta_kubernetes_pod_label_k8s_app

3

Thanks for the reply! Does that rule apply for the regex value too?

4

@petewall thank you so much, that was it! It’s all working now.

5

Yay! Glad that’s working for you now!