How are metrics series calculated?

Grafana Cloud
1

Hi,

I’m currently on the free Grafana cloud plan. It says there’s a metric limit of 10k metric series. My 2 small K3s clusters have already occupied over 5.5k metrics on the Metrics billing dashboard. Here are some questions I have:

  1. Do these metrics increase overtime as the clusters keep running or do they only increase when a new cluster is added?
  2. What happens if I exceed my 10k metrics limit? Will I not get any more metrics streamed to the grafana cloud?
  3. I need metrics to view the status of Kubernetes objects in my cluster and also to know how much memory and CPU is being consumed by the pods and how much resources are left on the node. Any way I can optimize the number of metrics being sent?
  4. If I upgrade to the pro plan on Grafana Cloud and exceed the limit of 20k, will I be charged extra without my approval?

Thanks a lot!

2

Hi @abdur

  1. Active Series would only increase if/when any new Metrics or label combinations are sent. In this case, when a new cluster is added, it likely would include a label indiciating the cluster and therefore would be an additional Series.
  2. On Grafana Cloud Free, that is correct: once you hit 10k active series it’ll start dropping any new Series.
  3. There a many ways to solve for this. These docs about using Grafana Cloud’s Kubernetes Monitoring service should help. By default, the Agent only will collect metrics displayed on dashboards/alerts. You’ll also see a series of Cardinality Dashboards (Dashboards → Cardinality) that will help you identify high-cardinality metrics (typically what drives the volume of metrics)
  4. On the Pro plan, we recommend creating alerts to be notified once you cross certain thresholds. The system is pay-as-you-go so there aren’t built-in limitations at the moment.
3

Thanks a lot @daviddorman! I’ll look into the resources you suggested.