No series on Grafana cloud even though Prometheus shows active metrics

Grafana Cloud
1

Hi all,

I’m testing the Grafana Cloud Agent on Kubernetes (K3s cluster). I installed the agent accourding to the Quickstart Guide, everything seems to be running on the cluster.

The agent seems to be working according to the logs and I see that in the stack page Prometheus shows 3084 active metrics as well.

However, if I go to Grafana Cloud it shows no metrics/series. I tried analyzing Prometheus metrics usage with Grafana Explore but it does not return any results.

image

If I query the api it does return metrics.

image
image1306×875 92 KB

I must be doing something wrong, might be at the scraper but i followed the instructions on the quick start.

Any thoughts?

Thanks in advance.

2

Hi @afermon!

Since you’re seeing 3084 metrics on the stack page, and can query the metrics using the API, it appears that they are in fact being shipped to the Grafana Cloud endpoint and are being stored there. Which makes me think it’s a Grafana-specific issue.

Can you double check that you’re querying against the correct datasource? (It should be the Grafana Cloud Prometheus metrics endpoint). What do you see in the billing/usage dashboard? Can you try a simpler query than the topk one you have above? What about just apiserver_request_total? Do you see any series when you query just for that?