I use private data source connect for connection from grafana cloud to my self-hosted prometheus in k8s cluster. It worked well for months but 3 days ago it suddenly stopped working. PDC agent is connected (as says grafana gui), nothing strange in logs (even in debug level). When I try to connect to configured data source, it returns this error:
Post "http://kube-prometheus-stack-prometheus.monitoring.svc.cluster.local:9090/api/v1/query": socks connect tcp private-datasource-connect.hosted-grafana.svc.cluster.local:443->kube-prometheus-stack-prometheus.monitoring.svc.cluster.local:9090: tls: failed to verify certificate: x509: certificate signed by unknown authority - There was an error returned querying the Prometheus API.
What’s strange is that another datasource from different PDC stopped working as well. Was there some breaking change in PDC service? What could I do? I even tried to create new PDC and new data source. All same - certificate verification error. Skip TLS verify option didn’t help.
Hosts in k8s cluster are ubuntu 24.04.2, ssh info: OpenSSH_9.6p1 Ubuntu-3ubuntu13.11, OpenSSL 3.0.13.