How to format tls certs in .yaml for Grafana datasource provisioning?

We’re trying to mimic the settings in the UI for datasource provisioning in our .yaml. However, we’re consistently getting errors:

"Failed to parse TLS CA PEM certificate"

We’ve tried (a) without the -----BEGIN CERTIFICATE-----, (b) a copy paste of the certificate, and (c) removing all \n.

Our .yaml has the following format:

secureJsonData:
     tlsCACert: "-----BEGIN CERTIFICATE-----
     MII[...]Lz
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MII[..]d4=
-----END CERTIFICATE-----"
     tlsClientCert: "-----BEGIN CERTIFICATE-----
MII[..]c+g==
-----END CERTIFICATE-----"
     tlsClientKey: "-----BEGIN RSA PRIVATE KEY-----
MI[..]6Kd
-----END RSA PRIVATE KEY-----"
2 Likes

Would like to make a slight bump on this – could we get some guidance on what format the .yaml expects when including our certificates / keys?

1 Like

You can use test data as an example:

3 Likes

For anyone else who encounters this - it looks like it’s related to the YAML formatting rather than the key. i got it to work looking like:

  jsonData: 
    tlsAuthWithCACert: true
  secureJsonData: 
    tlsCaCert: | 
      -----BEGIN CERTIFICATE-----
      line1
      line2
      .....
      ......
      -----END CERTIFICATE-----
5 Likes

Thanks @wrosscopeeko – would be super if someone from Grafana could update the docs to be better aligned with this example.

1 Like

What documentation from Grafana says xyz format is required for certificates. I see none.

1 Like

So it is good opportunity on your side to fix it. Doc is maintained via GitHub and it provides also link, where you can edit&propose doc change:

1 Like

Sure, once I find the answer for the questions I have.

1 Like

PEM formatted string?

PEM is defined in RFCs 1421 through 1424, this is a container format that may include just the public certificate (such as with Apache installs, and CA certificate files /etc/ssl/certs ), or may include an entire certificate chain including public key, private key, and root certificates. Confusingly, it may also encode a CSR (e.g. as used here) as the PKCS10 format can be translated into PEM. The name is from Privacy Enhanced Mail (PEM), a failed method for secure email but the container format it used lives on, and is a base64 translation of the x509 ASN.1 keys.

Cited from https://serverfault.com/questions/9708/what-is-a-pem-file-and-how-does-it-differ-from-other-openssl-generated-key-file

That PEM string can be multiline string, so then you need to know how to write multiline string in YAML. That also has own rules https://yaml-multiline.info/

2 Likes

Thank @wrosscopeeko for providing the solution to this (wrong documentation) issue on how to add the CA.

Maybe we should add also here for future reference to other users that the CA on your example has a single white character added on all the CA lines.

It is not easy to understand from the text that the user needs to add also whitespaces until that point.

To add complete description:

# config file version
apiVersion: 1

sioning/#data-sources
datasources:
  - name: Sample
    # <string, required> datasource type. Required
    type: prometheus
    # <string, required> access mode. proxy or direct (Server or Browser in the UI). Required
    access: proxy
    # <int> org id. will default to orgId 1 if not specified
    orgId: 1
    basicAuth: true
    # <string> basic auth username, if used
    basicAuthUser: {{ user }}
    # <bool> enable/disable with credentials headers
    withCredentials: true
    # <bool> mark as default datasource. Max one per org
    isDefault: true
    # <map> fields that will be converted to json and stored in json_data
    jsonData:
      graphiteVersion: "1.1"
      tlsAuth: false
      tlsAuthWithCACert: true
      oauthPassThru: true
    # <string> json object of data that will be encrypted.
    secureJsonData:
      tlsCACert: |2
        -----BEGIN CERTIFICATE-----
        line 1
        line 2
        .
        .
        .
        -----END CERTIFICATE-----

The interesting part to notice here is |<int> this indicates how many white space lines is expected after the pipe. By default (ommited) is 1. Anything above that (if desired) needs to be defined.

In case that someone wants to add the CA / CRT or KEY through Ansible the user can use Jinja2 with templates.

I managed to achieve it by loading the CA / CRT and KEY into vars and then adding it on the file.

Sample of code:

- name: Append 8 white space characters on ca                                                                                                                                                                                                  
  register: grafanaCa
  ansible.builtin.shell:
    cmd: "sed 's/^/        /g' {{ role_path }}/files/{{ grafana.cert.files.ca }}"
  no_log: true

And then on the relevant file that the user wants to update:

datasources:
  # <string, required> name of the datasource. Required
  - name: Prometheus
    # <string, required> datasource type. Required
    type: prometheus
    # <string, required> access mode. proxy or direct (Server or Browser in the UI). Required
    access: proxy
    # <int> org id. will default to orgId 1 if not specified
    orgId: 1
    # <string, required> access mode. proxy or direct (Server or Browser in the UI). Required
    url: https://{{ hostname }}:{{ prometheus.conf.port }}
    # <string> database user, if used
    user:
    # <string> database name, if used
    database:
    # <bool> enable/disable basic auth
    basicAuth: true
    # <string> basic auth username, if used
    basicAuthUser: {{ user }}
    # <bool> enable/disable with credentials headers
    withCredentials: true
    # <bool> mark as default datasource. Max one per org
    isDefault: true
    # <map> fields that will be converted to json and stored in json_data
    jsonData:
      graphiteVersion: "1.1"
      tlsAuth: true
      tlsAuthWithCACert: true
      oauthPassThru: true
    # <string> json object of data that will be encrypted.
    secureJsonData:
      tlsCACert: |2
{{ grafanaCa.stdout }}
      tlsClientCert: |2
{{ grafanaCrt.stdout }}
      tlsClientKey: |2
{{ grafanaKey.stdout }}

Hope this helps someone else as I spent way too much time trying to figure it out…

1 Like

Spent the whole afternoon bashing my head against this… Thanks @thanosegw, you’re awesome!

My solution for using a local cert on the ansible controller:

{% set ca_file_contents = lookup('file', '/local/path/ca.crt') %}

datasources:
  - name: Prometheus
    ...
    secureJsonData:
      tlsCACert: |
        {{ ca_file_contents | replace("\n", "\n        ") }}