I have deployed grafana helm.sh/chart=grafana-8.9.0. Its installed in a dashoarding namespace. Similarly, I have installed victoria-metrics in a different namespace victoria-metrics.
I have deployed istio with sidecar enabled in both namespaces. I have also configured peerauthentication in both namespaces with mtls.mode: STRICT.
Now when I try to configure the victoria-metrics datasource in grafana, I get the following error:
Post "https://victoria-metrics-victoria-metrics-cluster-vmselect.victoria-metrics.svc.cluster.local:8481/select/multitenant/prometheus": tls: failed to verify certificate: x509: certificate is not valid for any names, but wanted to match victoria-metrics-victoria-metrics-cluster-vmselect.victoria-metrics.svc.cluster.local - There was an error returned querying the Prometheus API.
However if I make the setting mtls.mode: PERMISSIVE in the victoria-metrics namespace, and change the datasource URL in grafana to HTTP (http://victoria-metrics-victoria-metrics-cluster-vmselect.victoria-metrics.svc.cluster.local:8481/select/multitenant/prometheus) it works.
So this defeats the purpose. how to properly configure datasources such as in this scenario where mTLS from the service mesh is enabled?
Any documentation / tutorial would be appreciated.
Thanks