PDC data source - TLS - x509: certificate signed by unknown authority

I use private data source connect for connection from grafana cloud to my self-hosted prometheus in k8s cluster. It worked well for months but 3 days ago it suddenly stopped working. PDC agent is connected (as says grafana gui), nothing strange in logs (even in debug level). When I try to connect to configured data source, it returns this error:

Post "http://kube-prometheus-stack-prometheus.monitoring.svc.cluster.local:9090/api/v1/query": socks connect tcp private-datasource-connect.hosted-grafana.svc.cluster.local:443->kube-prometheus-stack-prometheus.monitoring.svc.cluster.local:9090: tls: failed to verify certificate: x509: certificate signed by unknown authority - There was an error returned querying the Prometheus API.

What’s strange is that another datasource from different PDC stopped working as well. Was there some breaking change in PDC service? What could I do? I even tried to create new PDC and new data source. All same - certificate verification error. Skip TLS verify option didn’t help.

Hosts in k8s cluster are ubuntu 24.04.2, ssh info: OpenSSH_9.6p1 Ubuntu-3ubuntu13.11, OpenSSL 3.0.13.

2 Likes

I observed the same error, using the PDC binary on Debian 11.11. No particular error visible when running PDC in verbose mode. Upgrading to the latest release (v0.0.42) didn’t change anything.
I got notified of the first occurrence on May 29 (17:39UTC).

Trace ID: 620a399ce052118ae1ae81faaf2b6e13

1 Like

Working again since yesterday ~17h UTC. Thanks to whoever fixed this!