Datasource mode server is not used for all queries

What happened:
Our remote Prometheus is running in AWS EKS (k8s cluster maintained by AWS) and we restrict access to that datasource to just Grafana server IP address.
Datasource is configured to use server access mode.
But it seems that some queries to Prometheus are send from client (browser) instead of server based on this error:
image

Also we can’t see any data in Metrics browser which is part of Explore feature:
image

BUT direct query like up == 1 returns valid data as this query is initiated from Grafana server and not from the client .

What you expected to happen:
All queries should reflect access setting and in our case all traffic should be initiated from Grafana server as we have server mode set.

How to reproduce it (as minimally and precisely as possible):

  1. Create Prometheus datasource with server access mode.
  2. Restrict access (iptables, firewall, …) to that Prometheus server just for Grafana server.
  3. Then use Explore feature in Grafana.

Do you see some data in Metrics browser or not?

Anything else we need to know?:
It seems that requests bellow are initiated from client and not from Grafana server :

  • POST /prometheus/api/v1/series
  • POST /prometheus/api/v1/labels
  • GET /prometheus/api/v1/metadata

Environment:

  • Grafana version: 9.1.1 and 8.4.5
  • Data source type & version: Prometheus (v2.35.0)
  • OS Grafana is installed on: CentOS Stream 8 via RPM package
  • User OS & Browser: Linux and Mac, Google Chrome
  • Grafana plugins: N/A
  • Others: N/A

Am I missing something? Is this behavior common?
I created for this bug report (ID 54213) but it was rejected as a question :slightly_frowning_face:, that’s why I’m here.