Reg: sending k8s application metrics to grafana cloud

Greetings! I have recently started exploring grafana cloud and have deployed the Kubernetes integrations by following the official docs, it works fine and the bundled dashboards are awesome! :+1:

Our applications (written in Go) also emit Prometheus metrics, Is there a way I can send this metrics to grafana cloud? so that I can visualize this in grafana cloud by adding a custom dashboard. thanks!

Hey @prasus thanks for your message!

If your app containers are publishing metrics at the standard /metrics endpoint, you have to create a scrape job for your apps, which will likely use Kubernetes service discovery. You can learn more here. You can find some sample job configs here.

For example, this may help you get started:

  - job_name: "kubernetes-pods"

    kubernetes_sd_configs:
      - role: pod

    relabel_configs:
      - action: labelmap
        regex: __meta_kubernetes_pod_label_(.+)
      - source_labels: [__meta_kubernetes_namespace]
        action: replace
        target_label: namespace
      - source_labels: [__meta_kubernetes_pod_name]
        action: replace
        target_label: pod

This will create scrape targets for each container port of each Pod (with default scrape path of /metrics). To filter out or specify custom ports / Pods / metric paths, you can use relabeling and annotations / Pod labels. Examples are provided in the links above. This stackoverflow answer may also be helpful.

If youโ€™re using Services and Endpoints, you can also automatically discover targets using those abstractions, too. See Service and Endpoint.

Hope this helps!

Thanks a lot @hjet, I will explore this.