Secrets management, the entire config as a secret?

Grafana Loki
, , ,
1

Looking through the values for S3 I realised that secrets have to be stored in clear text:

loki:
  storage:
    type: s3
    s3:
      endpoint: "minio.local:9000"
      s3ForcePathStyle: true
      accessKeyId: "secretId"
      secretAccessKey: "secretKey"

I’m using gitops (Flux CD) to manage my cluster, so I don’t want to push secrets in plain text to git, and the values are then stored in a configmap in clear text after being installed.

I found this option, that would allow me do save the whole of config.yaml as a secret and mount it:

loki:
  existingSecretForConfig: "secretConfig"

Is there not a way to use kubernetes secrets for the few fields in the configuration?

Maybe new fields, with the name of a file containing the secret?

loki:
  storage:
    type: s3
    s3:
      endpoint: "minio.local:9000"
      s3ForcePathStyle: true
      accessKeyIdFile: "/secrets/s3/accessKeyId"
      secretAccessKeyFile: "/secrets/s3/secretAccessKey"

Can I use environment variables in loki’s configuration?
Something like this?

loki:
  storage:
    type: s3
    s3:
      endpoint: "minio.local:9000"
      s3ForcePathStyle: true
      accessKeyId: "${S3_KEY_ID}"
      secretAccessKey: "${S3_ACCESS_KEY}"

So that the configmap looks like this?

apiVersion: v1
kind: ConfigMap
data:
  config.yaml: |
    auth_enabled: false
    common:
      compactor_address: 'loki-read'
      path_prefix: /var/loki
      replication_factor: 3
      storage:
        s3:
          access_key_id: "${S3_KEY_ID}"
          bucketnames: loki-chunks
          endpoint: minio.local:9000
          s3forcepathstyle: true
          secret_access_key: "${S3_ACCESS_KEY}"
1 Like
2

As I am using Flux, I think that I have found an solution. Mozilla SOPS.

I’ve added the following .sops.yaml file to the repository:

creation_rules:
  - encrypted_regex: ^(accessKeyId|secretAccessKey)$
    age: >-
      age1myPublicKey,
      age1serverPublicKey

which encodes the contents of the 2 values for the server to ingest:

                s3:
                    endpoint: minio.local:9000
                    s3ForcePathStyle: true
                    insecure: true
                    accessKeyId: ENC[AES256_GCM,data:....,type:str]
                    secretAccessKey: ENC[AES256_GCM,data:....,type:str]

The values are decoded before the config map is generated. It still isn’t the greatest of solutions but it keeps the secrets safe in git.

3

Looks like I have done this with env vars and standard secrets, so should work.

     aws:
     s3: s3://${LOKI_S3_ACCESS_KEY_ID}:${LOKI_S3_SECRET_ACCESS_KEY}@my-region/my-bucket

On my deployment I can see this

          env:
            - name: LOKI_S3_ACCESS_KEY_ID
              valueFrom:
                secretKeyRef:
                  key: loki-s3-rw-access-key
                  name: loki-s3
            - name: LOKI_S3_SECRET_ACCESS_KEY
              valueFrom:
                secretKeyRef:
                  key: loki-s3-rw-secret-key
                  name: loki-s3

This was setup about a year ago and not changed since so not sure if something would have changed…

1 Like
4

Excellent, so I can use environment variables. Thanks

They aren’t resolved in the config map are they? They are still ${…} place holders?

5

One thing I had forgotten about… You have to configure -config.expand-env=true

Configmaps just have the placeholders, yes.

1 Like
6

Is it possible to use an IAM role and add that to the service account versus having to use ACCESS_KEY_ID / SECRET_ACCESS_KEY

7

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.