Default k8s agent/agent-loki configs exceed free usage limit

Grafana Cloud
1

Hello,

we’ve installed the default k8s configs agent.yaml and agent-loki.yaml from agent/production/kubernetes at main · grafana/agent · GitHub into a small k8s cluster (3 nodes, about 20 pods). Everything works well, except that we’re exceeding the free tier usage limits, currently being at 13.4k metrics.

Do different config files exist that filter the metrics to stay within the free tier limits, or do we have to implement the filtering ourselves using something like /docs/grafana-cloud/how-do-i/control-prometheus-metrics-usage/usage-reduction-k8s/?

Any help, including linking to an improved config somewhere else, would be appreciated.

Kind regards,
Fiete

2

Hi Fiete,

Yes, that config does scrape a lot by default. For a more lightweight config, you can try the one provisioned for you as part of the Kubernetes integration. Click on the lightning bolt from your hosted Grafana instance, then Kubernetes. It’ll give you an install script and ConfigMap for the Grafana Agent that only scrapes the cadvisor and kubelet endpoints on your Nodes, which will lower your usage. You’ll also get a handy set of dashboards at the same time!

To reduce further will require some tinkering. The resources in Control Prometheus metrics usage | Grafana Labs should be helpful for this.

Hope this helps!

3

Note that ConfigMap is for metrics only, you’ll still need a separate Agent DaemonSet and ConfigMap for logs: agent-loki.yaml in agent/production/kubernetes at main · grafana/agent · GitHub which you can install using a script: agent/install-loki.sh at main · grafana/agent · GitHub