AKS Prometheus 403 Access denied

Grafana Configuration
1

I installed Prometheus and Grafana on Azure Kubernetes Service, added ingresses so I can remotely access both of them. Prometheus displays graphs nicely (with the exception of apiserver stuff, which is not available on AKS since it’s managed).

I can access Grafana as well, but almost all queries yield an “403 Access denied” result. Only pod networking is shown.

I now (accidentally) removed the preinstalled Prometheus data source by adding a new org and removing the main org. When I try to add a new data source I also get “403 Access denied” on save. I’m trying to connect to http://kube-prometheus:9090 with proxy, no auth - which works when curling from grafana pod.

What is causing this error and how can I debug/resolve this issue? Grafana pod logs show only information from a few hours (grafana) or even days (gafana-watcher) ago…

2

While trying to debug the issue, i found out that when i remove params about container or container_name from API calls, they succeed.

However, when I try to save the changes, POSTing to /api/dashboards/db/ fails again with 403, so config can’t be persisted. Any hints?