I installed Prometheus and Grafana on Azure Kubernetes Service, added ingresses so I can remotely access both of them. Prometheus displays graphs nicely (with the exception of apiserver stuff, which is not available on AKS since it’s managed).
I can access Grafana as well, but almost all queries yield an “403 Access denied” result. Only pod networking is shown.
I now (accidentally) removed the preinstalled Prometheus data source by adding a new org and removing the main org. When I try to add a new data source I also get “403 Access denied” on save. I’m trying to connect to http://kube-prometheus:9090 with proxy, no auth - which works when curling from grafana pod.
What is causing this error and how can I debug/resolve this issue? Grafana pod logs show only information from a few hours (grafana) or even days (gafana-watcher) ago…