Grafana Cloud fails on a valid Let's Encrypt certificate endpoint

Grafana Configuration
1
  • What Grafana version and what operating system are you using?

Grafana Cloud (grafana dot net)

  • What are you trying to achieve?

Scrape metrics from an OpenWRT server running prometheus-node-exporter-lua behind nginx to provide TLS and basic auth as required by Grafana Cloud.

  • How are you trying to achieve it?

NOTE: I use domain.com and prometheus.domain.com to protect my server’s identity. I’d be happy to provide the real host names in private if it can help support.

Installed prometheus-node-exporter-lua and nginx
Configured nginx with:

config main global
	option uci_enable 'true'

config server '_lan'
	list listen '12345 ssl default_server'
	list include 'conf.d/*.locations'
	option ssl_certificate '/etc/acme/domain.com/domain.com.cer'
	option ssl_certificate_key '/etc/acme/domain.com/domain.com.key'
	option ssl_session_cache 'shared:SSL:32k'
	option ssl_session_timeout '64m'
	option access_log 'on; # logd openwrt'

nginx proxy config to forward from https on port 12345 to the Prometheus server on internal port 9100 on localhost:

server {
	listen 12345 ssl http2;
	server_name prometheus.domain.com;
	ssl_certificate /etc/acme/domain.com/domain.com.cer;
	ssl_certificate_key /etc/acme/domain.com/domain.com.key;
	ssl_session_cache shared:SSL:32k;
	ssl_session_timeout 64m;
	access_log on;
	auth_basic "Metrics";
	auth_basic_user_file /etc/nginx/conf.d/openwrt.passwd;
	location /metrics {
		proxy_pass http://localhost:9100/metrics;
		proxy_set_header HOST $host;
		proxy_set_header X-Forwarded-Scheme $scheme;
		proxy_set_header X-Forwarded-Proto $scheme;
		proxy_set_header X-Real-IP $remote_addr;
		proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
	}
}

Setup a username and password for basic auth.

  • What happened?

When accessing the URL from my laptop over local network, laptop over VPN to simulate external access or from my phone over mobile, curl or wget, I have no problem seeing the metrics from the Prometheus server without any warnings or errors from the client.

But when I try to configure the metrics endpoint in Grafana Cloud (Home > Connections > Add new connection > Metrics Endpoint)

and click Test Connection I get:

Screenshot 2025-04-09 at 8.31.55 am
Screenshot 2025-04-09 at 8.31.55 am755×127 10.7 KB

  • What did you expect to happen?

The “Test Connection” to pass and the endpoint to be configured.

  • Can you copy/paste the configuration(s) that you are having problems with?

Done above.

  • Did you receive any errors in the Grafana UI or in related logs? If so, please tell us exactly what they were.

The request to the provided endpoint did not return a valid response. Please ensure the https certificate is valid.
For help troubleshooting common errors see our documentation.

About the certificate on prometheus.domain.com:

  • It’s issued by Let’s Encrypt using acme.sh script on the OpenWRT box
  • It doesn’t get complaints from any other client I tried (Chrome browser on MacOS, curl, wget or browsers on my phone through mobile connection)
  • Its Subject is CN=domain.com and Subject Alternative Name are: DNS:*.domain.com and DNS:domain.com
  • It does NOT have DNS:prometheus.domain.com explicitly mentioned in it, and my attempt to try to add it broke the ACME process so I reverted.
  • So to clarify - the TLS certificate validity depends on the wildcard *.domain.com being accepted by the Grafana Cloud scraper.
2

To test what’s wrong, I ran Grafana inside a Docker image on my laptop and configured it to try to access my Prometheus scraping point and got:

Post "https://prometheus.domain.com:12345/metrics/api/v1/query": tls: failed to verify certificate: x509: certificate signed by unknown authority - There was an error returned querying the Prometheus API.

So, assuming the Grafana Docker image uses the same set of Roota CA’s as Grafana Cloud - does it recognise Let’s Encrypt CA?

Is the “ISRG Root X1” certificate recognised by Grafana Cloud?

3

I believe you have incorrect TLS, Please provide test result of your endpoint from GitHub - testssl/testssl.sh: Testing TLS/SSL encryption anywhere on any port

4

Thanks for getting back to me.
The results of the test are copied below (sans the hostname and IP address of my host).

In the meantime, I fired up Grafana in Docker on my laptop and discovered that it was trying to access .../api/v1/query attached to the URL I provide it, which taught me that it expects a Prometheus endpoint, not a “raw metrics” endpoint which returns things like:

# TYPE node_scrape_collector_success gauge
# TYPE node_nf_conntrack_entries gauge
node_nf_conntrack_entries 654
# TYPE node_nf_conntrack_entries_limit gauge
node_nf_conntrack_entries_limit 64512
node_scrape_collector_duration_seconds{collector="conntrack"} 0.0018692016601562
node_scrape_collector_success{collector="conntrack"} 1
# TYPE node_boot_time_seconds gauge
node_boot_time_seconds 1743309252
...

I have since configured the Docker-based Prometheus to scrape the metrics from the OpenWRT, then configured the Docker-based Grafana to use it for dashboards.

I’m working on repeating this using Grafana Cloud now.

My question now is: Can Cloud Grafana scrape the raw metrics directly? Or do I have to have Prometheus or something equivalent between the raw metrics and Cloud Grafana?

The motivation is to reduce the number of moving parts required for monitoring in my home network to the minimum.

Here are the test results:

 Service detected:       HTTP

 Testing protocols via sockets except NPN+ALPN

 SSLv2      not offered (OK)
 SSLv3      not offered (OK)
 TLS 1      not offered
 TLS 1.1    not offered
 TLS 1.2    offered (OK)
 TLS 1.3    offered (OK): final
 NPN/SPDY   not offered
 ALPN/HTTP2 h2, http/1.1 (offered)

 Testing cipher categories

 NULL ciphers (no encryption)                      not offered (OK)
 Anonymous NULL Ciphers (no authentication)        not offered (OK)
 Export ciphers (w/o ADH+NULL)                     not offered (OK)
 LOW: 64 Bit + DES, RC[2,4], MD5 (w/o export)      not offered (OK)
 Triple DES Ciphers / IDEA                         not offered
 Obsoleted CBC ciphers (AES, ARIA etc.)            offered
 Strong encryption (AEAD ciphers) with no FS       offered (OK)
 Forward Secrecy strong encryption (AEAD ciphers)  offered (OK)


 Testing server's cipher preferences

Hexcode  Cipher Suite Name (OpenSSL)       KeyExch.   Encryption  Bits     Cipher Suite Name (IANA/RFC)
-----------------------------------------------------------------------------------------------------------------------------
SSLv2
 -
SSLv3
 -
TLSv1
 -
TLSv1.1
 -
TLSv1.2 (no server order, thus listed by strength)
 xc030   ECDHE-RSA-AES256-GCM-SHA384       ECDH 521   AESGCM      256      TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
 xc028   ECDHE-RSA-AES256-SHA384           ECDH 521   AES         256      TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384
 xc014   ECDHE-RSA-AES256-SHA              ECDH 521   AES         256      TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
 xcca8   ECDHE-RSA-CHACHA20-POLY1305       ECDH 521   ChaCha20    256      TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256
 x9d     AES256-GCM-SHA384                 RSA        AESGCM      256      TLS_RSA_WITH_AES_256_GCM_SHA384
 xc0a1   AES256-CCM8                       RSA        AESCCM8     256      TLS_RSA_WITH_AES_256_CCM_8
 xc09d   AES256-CCM                        RSA        AESCCM      256      TLS_RSA_WITH_AES_256_CCM
 x3d     AES256-SHA256                     RSA        AES         256      TLS_RSA_WITH_AES_256_CBC_SHA256
 x35     AES256-SHA                        RSA        AES         256      TLS_RSA_WITH_AES_256_CBC_SHA
 xc02f   ECDHE-RSA-AES128-GCM-SHA256       ECDH 521   AESGCM      128      TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
 xc027   ECDHE-RSA-AES128-SHA256           ECDH 521   AES         128      TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
 xc013   ECDHE-RSA-AES128-SHA              ECDH 521   AES         128      TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
 xc0a0   AES128-CCM8                       RSA        AESCCM8     128      TLS_RSA_WITH_AES_128_CCM_8
 xc09c   AES128-CCM                        RSA        AESCCM      128      TLS_RSA_WITH_AES_128_CCM
 x9c     AES128-GCM-SHA256                 RSA        AESGCM      128      TLS_RSA_WITH_AES_128_GCM_SHA256
 x3c     AES128-SHA256                     RSA        AES         128      TLS_RSA_WITH_AES_128_CBC_SHA256
 x2f     AES128-SHA                        RSA        AES         128      TLS_RSA_WITH_AES_128_CBC_SHA
TLSv1.3 (no server order, thus listed by strength)
 x1302   TLS_AES_256_GCM_SHA384            ECDH 253   AESGCM      256      TLS_AES_256_GCM_SHA384
 x1303   TLS_CHACHA20_POLY1305_SHA256      ECDH 253   ChaCha20    256      TLS_CHACHA20_POLY1305_SHA256
 x1301   TLS_AES_128_GCM_SHA256            ECDH 253   AESGCM      128      TLS_AES_128_GCM_SHA256

 Has server cipher order?     no (NOT ok)
 (limited sense as client will pick)

 Testing robust forward secrecy (FS) -- omitting Null Authentication/Encryption, 3DES, RC4

 FS is offered (OK)           TLS_AES_256_GCM_SHA384 TLS_CHACHA20_POLY1305_SHA256 ECDHE-RSA-AES256-GCM-SHA384 ECDHE-RSA-AES256-SHA384 ECDHE-RSA-AES256-SHA
                              ECDHE-RSA-CHACHA20-POLY1305 TLS_AES_128_GCM_SHA256 ECDHE-RSA-AES128-GCM-SHA256 ECDHE-RSA-AES128-SHA256 ECDHE-RSA-AES128-SHA
 Elliptic curves offered:     prime256v1 secp384r1 secp521r1 X25519 X448
 Finite field group:          ffdhe2048 ffdhe3072 ffdhe4096 ffdhe6144 ffdhe8192
 TLS 1.2 sig_algs offered:    RSA-PSS-RSAE+SHA512 RSA-PSS-RSAE+SHA384 RSA-PSS-RSAE+SHA256 RSA+SHA512 RSA+SHA384 RSA+SHA256 RSA+SHA224
 TLS 1.3 sig_algs offered:    RSA-PSS-RSAE+SHA512 RSA-PSS-RSAE+SHA384 RSA-PSS-RSAE+SHA256

 Testing server defaults (Server Hello)

 TLS extensions (standard)    "server name/#0" "max fragment length/#1" "supported_groups/#10" "EC point formats/#11" "application layer protocol negotiation/#16"
                              "encrypt-then-mac/#22" "extended master secret/#23" "session ticket/#35" "supported versions/#43" "key share/#51"
                              "renegotiation info/#65281"
 Session Ticket RFC 5077 hint 3840 seconds, session tickets keys seems to be rotated < daily
 SSL Session ID support       yes
 Session Resumption           Tickets: yes, ID: yes
 TLS clock skew               Random values, no fingerprinting possible
 Certificate Compression      none
 Client Authentication        none
 Signature Algorithm          SHA256 with RSA
 Server key size              RSA 2048 bits (exponent is 65537)
 Server key usage             Digital Signature, Key Encipherment
 Server extended key usage    TLS Web Server Authentication, TLS Web Client Authentication
 Serial                       0542262ADE4F2FDCE099BABAE5D9D47CEA31 (OK: length 18)
 Fingerprints                 SHA1 3CDA985FE44F5D76EE9F92C649ECCBEDAF32F448
                              SHA256 A7AD9DDAA4988C69741CC0917F02636C2D511DA926807BA3FE3FE129521F0882
 Common Name (CN)             domain.com
 subjectAltName (SAN)         *.domain.com domain.com
 Trust (hostname)             Ok via SAN wildcard (same w/o SNI)
                              wildcard certificate could be problematic, see other hosts at
                              https://search.censys.io/search?resource=hosts&virtual_hosts=INCLUDE&q=A7AD9DDAA4988C69741CC0917F02636C2D511DA926807BA3FE3FE129521F0882
 Chain of trust               NOT ok (chain incomplete)
 EV cert (experimental)       no
 Certificate Validity (UTC)   88 >= 30 days (2025-04-08 07:54 --> 2025-07-07 07:54)
 ETS/"eTLS", visibility info  not present
 Certificate Revocation List  http://r10.c.lencr.org/50.crl
 OCSP URI                     http://r10.o.lencr.org
 OCSP stapling                not offered
 OCSP must staple extension   --
 DNS CAA RR (experimental)    not offered
 Certificate Transparency     yes (certificate extension)
 Certificates provided        1
 Issuer                       R10 (Let's Encrypt from US)
 Intermediate Bad OCSP (exp.) Ok


 Testing HTTP header response @ "/metrics"

 HTTP Status Code             401 Unauthorized  WWW-Authenticate: Basic realm="Metrics"
 HTTP clock skew              -8 sec from localtime
 Strict Transport Security    not offered
 Public Key Pinning           --
 Server banner                nginx/1.25.0
 Application banner           --
 Cookie(s)                    (none issued at "/metrics") -- HTTP status 401 signals you maybe missed the web application
 Security headers             --
 Reverse Proxy banner         --


 Testing vulnerabilities

 Heartbleed (CVE-2014-0160)                not vulnerable (OK), no heartbeat extension
 CCS (CVE-2014-0224)                       not vulnerable (OK)
 Ticketbleed (CVE-2016-9244), experiment.  not vulnerable (OK)
 ROBOT                                     not vulnerable (OK)
 Secure Renegotiation (RFC 5746)           supported (OK)
 Secure Client-Initiated Renegotiation     not vulnerable (OK)
 CRIME, TLS (CVE-2012-4929)                not vulnerable (OK)
 BREACH (CVE-2013-3587)                    no gzip/deflate/compress/br HTTP compression (OK)  - only supplied "/metrics" tested
 POODLE, SSL (CVE-2014-3566)               not vulnerable (OK), no SSLv3 support
 TLS_FALLBACK_SCSV (RFC 7507)              No fallback possible (OK), no protocol below TLS 1.2 offered
 SWEET32 (CVE-2016-2183, CVE-2016-6329)    not vulnerable (OK)
 FREAK (CVE-2015-0204)                     not vulnerable (OK)
 DROWN (CVE-2016-0800, CVE-2016-0703)      not vulnerable on this host and port (OK)
                                           make sure you don't use this certificate elsewhere with SSLv2 enabled services, see
                                           https://search.censys.io/search?resource=hosts&virtual_hosts=INCLUDE&q=A7AD9DDAA4988C69741CC0917F02636C2D511DA926807BA3FE3FE129521F0882
 LOGJAM (CVE-2015-4000), experimental      not vulnerable (OK): no DH EXPORT ciphers, no DH key detected with <= TLS 1.2
 BEAST (CVE-2011-3389)                     not vulnerable (OK), no SSL3 or TLS1
 LUCKY13 (CVE-2013-0169), experimental     potentially VULNERABLE, uses cipher block chaining (CBC) ciphers with TLS. Check patches
 Winshock (CVE-2014-6321), experimental    not vulnerable (OK)
 RC4 (CVE-2013-2566, CVE-2015-2808)        no RC4 ciphers detected (OK)


 Running client simulations (HTTP) via sockets

 Browser                      Protocol  Cipher Suite Name (OpenSSL)       Forward Secrecy
------------------------------------------------------------------------------------------------
 Android 6.0                  TLSv1.2   ECDHE-RSA-AES128-GCM-SHA256       256 bit ECDH (P-256)
 Android 7.0 (native)         TLSv1.2   ECDHE-RSA-AES128-GCM-SHA256       256 bit ECDH (P-256)
 Android 8.1 (native)         TLSv1.2   ECDHE-RSA-AES128-GCM-SHA256       253 bit ECDH (X25519)
 Android 9.0 (native)         TLSv1.3   TLS_AES_128_GCM_SHA256            253 bit ECDH (X25519)
 Android 10.0 (native)        TLSv1.3   TLS_AES_128_GCM_SHA256            253 bit ECDH (X25519)
 Android 11 (native)          TLSv1.3   TLS_AES_128_GCM_SHA256            253 bit ECDH (X25519)
 Android 12 (native)          TLSv1.3   TLS_AES_128_GCM_SHA256            253 bit ECDH (X25519)
 Chrome 79 (Win 10)           TLSv1.3   TLS_AES_128_GCM_SHA256            253 bit ECDH (X25519)
 Chrome 101 (Win 10)          TLSv1.3   TLS_AES_128_GCM_SHA256            253 bit ECDH (X25519)
 Firefox 66 (Win 8.1/10)      TLSv1.3   TLS_AES_128_GCM_SHA256            253 bit ECDH (X25519)
 Firefox 100 (Win 10)         TLSv1.3   TLS_AES_128_GCM_SHA256            253 bit ECDH (X25519)
 IE 6 XP                      No connection
 IE 8 Win 7                   No connection
 IE 8 XP                      No connection
 IE 11 Win 7                  TLSv1.2   ECDHE-RSA-AES256-SHA384           256 bit ECDH (P-256)
 IE 11 Win 8.1                TLSv1.2   ECDHE-RSA-AES256-SHA384           256 bit ECDH (P-256)
 IE 11 Win Phone 8.1          TLSv1.2   AES128-SHA256                     No FS
 IE 11 Win 10                 TLSv1.2   ECDHE-RSA-AES256-GCM-SHA384       256 bit ECDH (P-256)
 Edge 15 Win 10               TLSv1.2   ECDHE-RSA-AES256-GCM-SHA384       253 bit ECDH (X25519)
 Edge 101 Win 10 21H2         TLSv1.3   TLS_AES_128_GCM_SHA256            253 bit ECDH (X25519)
 Safari 12.1 (iOS 12.2)       TLSv1.3   TLS_CHACHA20_POLY1305_SHA256      253 bit ECDH (X25519)
 Safari 13.0 (macOS 10.14.6)  TLSv1.3   TLS_CHACHA20_POLY1305_SHA256      253 bit ECDH (X25519)
 Safari 15.4 (macOS 12.3.1)   TLSv1.3   TLS_AES_128_GCM_SHA256            253 bit ECDH (X25519)
 Java 7u25                    No connection
 Java 8u161                   TLSv1.2   ECDHE-RSA-AES256-SHA384           256 bit ECDH (P-256)
 Java 11.0.2 (OpenJDK)        TLSv1.3   TLS_AES_128_GCM_SHA256            256 bit ECDH (P-256)
 Java 17.0.3 (OpenJDK)        TLSv1.3   TLS_AES_256_GCM_SHA384            253 bit ECDH (X25519)
 go 1.17.8                    TLSv1.3   TLS_AES_128_GCM_SHA256            253 bit ECDH (X25519)
 LibreSSL 2.8.3 (Apple)       TLSv1.2   ECDHE-RSA-CHACHA20-POLY1305       253 bit ECDH (X25519)
 OpenSSL 1.0.2e               TLSv1.2   ECDHE-RSA-AES256-GCM-SHA384       256 bit ECDH (P-256)
 OpenSSL 1.1.0l (Debian)      TLSv1.2   ECDHE-RSA-AES256-GCM-SHA384       253 bit ECDH (X25519)
 OpenSSL 1.1.1d (Debian)      TLSv1.3   TLS_AES_256_GCM_SHA384            253 bit ECDH (X25519)
 OpenSSL 3.0.3 (git)          TLSv1.3   TLS_AES_256_GCM_SHA384            253 bit ECDH (X25519)
 OpenSSL 3.5.0 (git)          TLSv1.3   TLS_AES_256_GCM_SHA384            253 bit ECDH (X25519)
 Apple Mail (16.0)            TLSv1.2   ECDHE-RSA-AES256-GCM-SHA384       256 bit ECDH (P-256)
 Thunderbird (91.9)           TLSv1.3   TLS_AES_128_GCM_SHA256            253 bit ECDH (X25519)


 Rating (experimental)

 Rating specs (not complete)  SSL Labs's 'SSL Server Rating Guide' (version 2009q from 2020-01-30)
 Specification documentation  https://github.com/ssllabs/research/wiki/SSL-Server-Rating-Guide
 Protocol Support (weighted)  0 (0)
 Key Exchange     (weighted)  0 (0)
 Cipher Strength  (weighted)  0 (0)
 Final Score                  0
 Overall Grade                T
 Grade cap reasons            Grade capped to T. Issues with the chain of trust (chain incomplete)
                              Grade capped to A. HSTS is not offered
5

That’s your TLS problem.

1 Like
6

Thanks. I see.
Could it be that the test script just doesn’t recognise the Let’s Encrypt CA Cert?
I’m asking because Chrome, curl and wget all accept the certificate without complaint.

7

No. Be familiar with how TLS verification works. Proper TLS verification needs to verify the full chain.

Check LE forum/doc - there are billions and billions of topics about not complete chain. That is a standard rookie issue.

8

Thanks for the pointers.
I got the SSL issue fixed by pointing nginx to the fullchain.cer file instead of the specific site chain and now I get:

 Chain of trust               Ok