How to configure HTTPS certificates for Kafka REST Proxy Alerts?

Grafana
1

Hi, I have the Grafana Alerts Kafka REST Proxy working for HTTP but get an error over HTTPS:
>> x509: “Kafka” certificate is not trusted
The curl command works without -k after I did this on my Mac so I know my certs file is good.

% cert-chain.pem >> /etc/ssl/cert.pem
curl -X POST \
     -H "Content-Type: application/vnd.kafka.json.v2+json" \
     -H "Accept: application/vnd.kafka.v2+json" \
     --data '{"records":[{"key":"jsmith","value":"alarm cows3"},{"key":"htanaka","value":"cows3"},{"key":"awalther","value":"bookshelves"}]}' \
     "https://myaddress:8082/topics/demo3"

I just can’t seem to get Grafana 9.3.1 to use the cert when posting the alerts. Can someone point me in the right direction?
Thanks,
Jim

2

Just add that “cert-chain.pem” to your OS CA certs, which are used by your Grafana.
Check doc of OS/orchestration which is running your Grafana for details how to do that.

3

I’ve added the cert-chain.pem CA Root Certs to my MacOS Keychain Access and it shows Trusted but I still get the same error. I’m not following you on “OS/orchestration doc”. I searched for orchestration and got nothing. Can you post a link to the doc? Thanks.

4

I tried adding my .pem content to
/usr/local/etc/ca-certificates/cert.pem
but that didn’t help.

5

No, I don’t know how is Grafana running.

E.g.

  • Grafana is running on Windows machine as binary → check windows doc how to add CA cert
  • Grafana is running on Windows machine as Docker Linux container → check docker doc how to add ca cert to the container

There is milion options how you can run Grafana and you didn’t provide any your details → you have generic recommendation.

6

Does it matter if I am using HTTP or HTTPS for my Grafana Webui Login? I’m using HTTP.

7

I changed to HTTPS and I still get this. A friend used docker-compose and the Grafana v10.1.6 image and it worked fine. I’m not having luck with my localhost MacOS v13.6.3 laptop running Grafana v9.3.1. Do you know if there was a fix between v9.3.1 and v10.1.6 that might account for this?

8

Here is the docker-compose test case that worked.
version: ‘3.8’
services:
grafana:
image: grafana/grafana-oss:10.1.6
container_name: grafana
restart: unless-stopped
# if you are running as root then set it to 0
# else find the right id with the id -u command
user: ‘501’
env_file:
- grafana.env
ports:
- ‘3000:3000’
volumes:
- /opt/docker/kafka_3.5.1/nginx/server.key:/var/lib/grafana/ssl/grafana.key:ro
- /opt/docker/kafka_3.5.1/nginx/server.pem:/var/lib/grafana/ssl/grafana.crt:ro
- /opt/docker/kafka_3.5.1/cert-chain.pem:/etc/ssl/certs/ca-certificates.pem:ro
# - /app/grafana:/var/lib/grafana

grafana.env
GF_SERVER_CERT_FILE=/var/lib/grafana/ssl/grafana.crt
GF_SERVER_CERT_KEY=/var/lib/grafana/ssl/grafana.key
GF_SERVER_PROTOCOL=https
GF_AUTH_BASIC_ENABLED=false

10

Which defaults.ini setting do I set for the ca_cert? The example I have only had these. Thanks.
[server]
cert_file = /etc/grafana/grafana.crt
cert_key = /etc/grafana/grafana.key

11

So your problem is:
How to add my custom CA cert (used by my custom Kafka REST Proxy) into my Grafana which is running in the Docker container?

12

Actually, my problem is that I am running on darwin (MacOS v13.6.3) and I can’t seem to get my known-good (since curl works with it) ca cert file to be used by Grafana. I’ve tried putting the file in serveral places and loading into the Mac Keychain Access but it is not working. I’m running Grafana locally with “./bin/grafana-server” and “./conf/defaults.ini”. How does Grafana on darwin load ca certs? It works fine from my mac using a linux docker container, but I’d like to run natively. Thanks. BTW, I see the same thing with Grafana 9.3.1 and 10.1.6 so its not the version of Grafana from what I can tell. It seems more like I’m not doing the right thing for the darwin target.

13

Hey, The certificate authority (CA) that issued the Kafka server certificate is trusted by Grafana. Grafana needs to recognize and trust the CA that signed the Kafka certificate. If necessary, add the CA certificate to the certificate bundle used by Grafana.