Grafana dashboard shows higher CPU usage from 11:00pm to 11:30pm

Grafana Prometheus
1

Hi,
I am using Grafana V7.5.16 and Prometheus 2.37.1 on Tanzu kubernetes cluster.
I have created a Grafana dashboard by following Promql -
100 - avg(irate(node_cpu_seconds_total{mode=“idle”}[5m]) * 100)

The CPU graph shows that the utilisation shoots up around 11:00pm and remain till 11:30pm approx everyday. I have checked the PODs utiliztion during that period. These are running fine without any shoots up.

I doubt that there is something at kubernetes cluster level but not able to find what is causing that.

Any idea how to find the root cause.

2

I am managed to figure out the issue of the CPU spikes. It was due to a scheduled jobs which was triggering at 11:00pm