Kubernetes Integration: CPU recording rule dashboards not working

olafkocanda · June 28, 2022, 3:46pm

Hi there,

I have been running the kubernetes integration quite smoothly before. Now after updating, it seems like none of the CPU metrics is working in the Dashboards.
queries like

sum(node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate{cluster="cloud"}) by (namespace)

are not receiving any data

Other queries like
sum(kube_node_status_capacity{cluster="cloud", node=~"raspberrypi", resource="cpu"})

seem odd since the scale is totally wrong (for a 4 cpu rpi):

but at least they receive some data:

I am able to receive all necessary metrics in the cloud but it seems like the recording rules are not working for CPU. I followed the manual for the kubernetes integration and even reinstalled it several times (including the kube state metrics deployment and grafana-agent statefulset).
I deleted and redeployed the configmap. However, it still does not want to work…

Any help or ideas what I am missing?

Melody · June 29, 2022, 8:50pm

Hello! I think the CPU metrics are gathered from the cAdvisor target, so if that is not present that could be a reason why.

If you haven’t already done so, would you please open a Support ticket so the team can take a closer look?

olafkocanda · June 30, 2022, 7:06am

Hey Melody,
I just checked again, the cadvisor job is integrated:

          job_name: integrations/kubernetes/cadvisor

I thought it might be solved easier in this forum but I guess I have to open a ticket.

Thanks!

ryangeyer · June 30, 2022, 5:33pm

Hello Olaf,

Not sure if you’ve submitted a support ticket yet (if so, feel free to DM me or post it here).

There are a couple other things to check here, first, see the attached screenshot. This shows verifying the existence of the recording rule. You can get to this UI by clicking the bell icon in the left nav (Alerting), then selecting the filter for Recording rules, and expanding the rule group highlighted.

If that’s there, the next thing we want to confirm is that the two metrics used in the rule are reporting correctly.

As Melody stated, one of those is a cAdvisor metric, and the other comes from kube-state-metrics (ksm). Given the other graph you showed, of a ksm metric, which was not displaying information as expected, I suspect you may not be running ksm, or it may be misconfigured.

How did you deploy ksm?

Are there any metrics populating with a job integrations/kubernetes/kube-state-metrics?

olafkocanda · June 30, 2022, 6:14pm

Hi ryangeyer,

thanks for coming back to me. I can confirm that the required recording rule exists:

I have all three integrations (cadvisor, kubelet and kube state metrics) for kubernetes running (see the kubernetes overview dashboard):

I deployed ksm via helm using the explanation in the Integrations and Connections → Kubernetes Monitoring part. As I had it running previously, I even uninstalled the existing, and went through all the steps again. I would like to look it up again, but, unfortunately, when clicking on Integrations and Connections the kubernetes monitoring integration (https://XXX.grafana.net/a/grafana-k8s-app/cluster) says it is not installed:
grafik
Further, I cannot open the panel “Kubernetes Monitoring”