The query (/api/v1/series) to Prometheus (HTTPS) in Variables doesn't send the Authorization


I set a query variable in a dashboard to get data from a https Prometheus. But it gets 401 since the query doesn’t send the Authorization to Prometheus.

The datasource works well when Grafana runs /api/v1/query. The access log from Prometheus is like - - monitoringuser [22/Dec/2022:15:05:10 +0000] "POST /api/v1/query?query=chrony_update_interval%7Bhost%3D%22%22%7D%2B0&time=1671721500 HTTP/1.1" 200 87 "-" "Grafana/9.3.2"

But when it comes to Variables, the access log is like - - [22/Dec/2022:15:05:24 +0000] "POST /api/v1/series HTTP/1.1" 401 381 "" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/ Safari/537.36" 

There is no username which means Grafana doesn’t send the Authorization, so it gets a 401 response.

However, for a http Prometheus, it works well calling /api/v1/series with an Authorization.

I even can call directly Grafana’s API -[]=system_uptime{}&start=1671675209&end=1671696809 and get response. But when I click “Run query” on the Variables page, it failed with 401.

Fixed the issue. The root cause is on a reverse proxy in the front of Prometheus which checks the HTTP_REFERER. The difference between a query in Variables and Dashboard is HTTP_REFERER. The HTTP_REFERER in a query of Variables is a URL of Grafana API, not a URL of Prometheus API.