I created multiple rules for many servers. The rules interval is 1 min.
After rules created, Grafana performance degraded obviously such as long loading time, unable to save dashboard with error “someone else has updated this dashboard”, API error.
Our loki-stack deployed on K8S. I found Grafana pod CPU increases with number of rules increase but the CPU utilization is still normal. After I remove a couple of rules, the performance is back to normal.
Would like to know how can I find the bottleneck?
Grafana version: 8.3.4