How to format tls certs in .yaml for Grafana datasource provisioning?

We’re trying to mimic the settings in the UI for datasource provisioning in our .yaml. However, we’re consistently getting errors:

"Failed to parse TLS CA PEM certificate"

We’ve tried (a) without the -----BEGIN CERTIFICATE-----, (b) a copy paste of the certificate, and (c) removing all \n.

Our .yaml has the following format:

secureJsonData:
     tlsCACert: "-----BEGIN CERTIFICATE-----
     MII[...]Lz
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MII[..]d4=
-----END CERTIFICATE-----"
     tlsClientCert: "-----BEGIN CERTIFICATE-----
MII[..]c+g==
-----END CERTIFICATE-----"
     tlsClientKey: "-----BEGIN RSA PRIVATE KEY-----
MI[..]6Kd
-----END RSA PRIVATE KEY-----"
2 Likes

Would like to make a slight bump on this – could we get some guidance on what format the .yaml expects when including our certificates / keys?

1 Like

You can use test data as an example:

3 Likes

For anyone else who encounters this - it looks like it’s related to the YAML formatting rather than the key. i got it to work looking like:

  jsonData: 
    tlsAuthWithCACert: true
  secureJsonData: 
    tlsCaCert: | 
      -----BEGIN CERTIFICATE-----
      line1
      line2
      .....
      ......
      -----END CERTIFICATE-----
5 Likes

Thanks @wrosscopeeko – would be super if someone from Grafana could update the docs to be better aligned with this example.

2 Likes

What documentation from Grafana says xyz format is required for certificates. I see none.

1 Like

So it is good opportunity on your side to fix it. Doc is maintained via GitHub and it provides also link, where you can edit&propose doc change:

1 Like

Sure, once I find the answer for the questions I have.

1 Like

PEM formatted string?

PEM is defined in RFCs 1421 through 1424, this is a container format that may include just the public certificate (such as with Apache installs, and CA certificate files /etc/ssl/certs ), or may include an entire certificate chain including public key, private key, and root certificates. Confusingly, it may also encode a CSR (e.g. as used here) as the PKCS10 format can be translated into PEM. The name is from Privacy Enhanced Mail (PEM), a failed method for secure email but the container format it used lives on, and is a base64 translation of the x509 ASN.1 keys.

Cited from certificate - What is a Pem file and how does it differ from other OpenSSL Generated Key File Formats? - Server Fault

That PEM string can be multiline string, so then you need to know how to write multiline string in YAML. That also has own rules https://yaml-multiline.info/

2 Likes

Thank @wrosscopeeko for providing the solution to this (wrong documentation) issue on how to add the CA.

Maybe we should add also here for future reference to other users that the CA on your example has a single white character added on all the CA lines.

It is not easy to understand from the text that the user needs to add also whitespaces until that point.

To add complete description:

# config file version
apiVersion: 1

sioning/#data-sources
datasources:
  - name: Sample
    # <string, required> datasource type. Required
    type: prometheus
    # <string, required> access mode. proxy or direct (Server or Browser in the UI). Required
    access: proxy
    # <int> org id. will default to orgId 1 if not specified
    orgId: 1
    basicAuth: true
    # <string> basic auth username, if used
    basicAuthUser: {{ user }}
    # <bool> enable/disable with credentials headers
    withCredentials: true
    # <bool> mark as default datasource. Max one per org
    isDefault: true
    # <map> fields that will be converted to json and stored in json_data
    jsonData:
      graphiteVersion: "1.1"
      tlsAuth: false
      tlsAuthWithCACert: true
      oauthPassThru: true
    # <string> json object of data that will be encrypted.
    secureJsonData:
      tlsCACert: |2
        -----BEGIN CERTIFICATE-----
        line 1
        line 2
        .
        .
        .
        -----END CERTIFICATE-----

The interesting part to notice here is |<int> this indicates how many white space lines is expected after the pipe. By default (ommited) is 1. Anything above that (if desired) needs to be defined.

In case that someone wants to add the CA / CRT or KEY through Ansible the user can use Jinja2 with templates.

I managed to achieve it by loading the CA / CRT and KEY into vars and then adding it on the file.

Sample of code:

- name: Append 8 white space characters on ca                                                                                                                                                                                                  
  register: grafanaCa
  ansible.builtin.shell:
    cmd: "sed 's/^/        /g' {{ role_path }}/files/{{ grafana.cert.files.ca }}"
  no_log: true

And then on the relevant file that the user wants to update:

datasources:
  # <string, required> name of the datasource. Required
  - name: Prometheus
    # <string, required> datasource type. Required
    type: prometheus
    # <string, required> access mode. proxy or direct (Server or Browser in the UI). Required
    access: proxy
    # <int> org id. will default to orgId 1 if not specified
    orgId: 1
    # <string, required> access mode. proxy or direct (Server or Browser in the UI). Required
    url: https://{{ hostname }}:{{ prometheus.conf.port }}
    # <string> database user, if used
    user:
    # <string> database name, if used
    database:
    # <bool> enable/disable basic auth
    basicAuth: true
    # <string> basic auth username, if used
    basicAuthUser: {{ user }}
    # <bool> enable/disable with credentials headers
    withCredentials: true
    # <bool> mark as default datasource. Max one per org
    isDefault: true
    # <map> fields that will be converted to json and stored in json_data
    jsonData:
      graphiteVersion: "1.1"
      tlsAuth: true
      tlsAuthWithCACert: true
      oauthPassThru: true
    # <string> json object of data that will be encrypted.
    secureJsonData:
      tlsCACert: |2
{{ grafanaCa.stdout }}
      tlsClientCert: |2
{{ grafanaCrt.stdout }}
      tlsClientKey: |2
{{ grafanaKey.stdout }}

Hope this helps someone else as I spent way too much time trying to figure it out…

2 Likes

Spent the whole afternoon bashing my head against this… Thanks @thanosegw, you’re awesome!

My solution for using a local cert on the ansible controller:

{% set ca_file_contents = lookup('file', '/local/path/ca.crt') %}

datasources:
  - name: Prometheus
    ...
    secureJsonData:
      tlsCACert: |
        {{ ca_file_contents | replace("\n", "\n        ") }}

Just another way of inserting tls secret data in datasources-provisioning.yaml, in case it can help somebody. In my case, I did not manage with ansible, Jinja2,… neither I wanted to install extra packages. I found an easy alternative way using ‘sed’ bash command (linux world).
It consists in preprocessing the file before grafana is started. You can do that in the Dockerfile, with RUN, or as a script that runs before the default init script of the image, every time the container is started.
In the latter case, as I did, I declared secrets, so I execute this code in the init script:

sed -i '/tlsClientCert/ r /run/secrets/gf_client_certificate' /etc/grafana/provisioning/datasources/datasources-provisioning.yaml
sed -i '/tlsCACert/ r /run/secrets/gf_ca_certificate' /etc/grafana/provisioning/datasources/datasources-provisioning.yaml
sed -i '/tlsClientKey/ r /run/secrets/gf_client_key' /etc/grafana/provisioning/datasources/datasources-provisioning.yaml

/run.sh

This is the initial content of datasources-provisioning.yaml (secure json zone only):

    secureJsonData:
      httpHeaderValue1: "Token ${GF_DATASOURCE_INFLUXDB_TOKEN}"
      tlsCACert: |
      tlsClientCert: |
      tlsClientKey: |

With the certs and key files well formatted (8 blanks at the beginning of each line), it works nicely. Hope this adds another posible solution for somebody