Storing s3 accessKeyID and secretAccessKey securely

Hi,

I’m deploying Loki with Grafana Community Kubernetes Helm Charts | helm-charts and using the following configuration for the S3 auth:

...
loki:
  storage:
    type: s3
    s3:
      accessKeyId: <grafana-loki-s3-acceskeyid>
      secretAccessKey: <grafana-loki-s3-secretaccesskey>
	...

My question is:
How can I read the values for accessKeyId and secretAccessKey from a k8s secret?

Supposing I have the following secret, how do I use it in the above example?:

apiVersion: v1
kind: Secret
metadata:
  name: loki-secrets
  namespace: grafana-loki
data:
  grafana-loki-s3-accessKeyId: **************
  grafana-loki-s3-secretAccessKey: **************
type: Opaque

TIA!

1 Like

You can do it a few different ways. It is pretty common to do something like this Define a container environment variable with data from a single Secret

And then you can Use environment variables in the configuration

1 Like

Thanks @b0b!

I have now something like this in my Helm values file for deploying Loki and it works nicely :slight_smile:

loki:
  ...
  storage:
    ...
    s3:
      endpoint: "${GRAFANA-LOKI-S3-ENDPOINT}"
      accessKeyId: "${GRAFANA-LOKI-S3-ACCESKEYID}"
      secretAccessKey: "${GRAFANA-LOKI-S3-SECRETACCESSKEY}"
      ...
backend:
  ...
  extraArgs:
    - '-config.expand-env=true'
  extraEnv:
    - name: GRAFANA-LOKI-S3-ENDPOINT 
      valueFrom:
        secretKeyRef:
          name: loki-secrets
          key: grafana-loki-s3-endpoint
    - name: GRAFANA-LOKI-S3-ACCESKEYID 
      valueFrom:
        secretKeyRef:
          name: loki-secrets
          key: grafana-loki-s3-accessKeyId
    - name: GRAFANA-LOKI-S3-SECRETACCESSKEY
      valueFrom:
        secretKeyRef:
          name: loki-secrets
          key: grafana-loki-s3-secretAccessKey
  ...

1 Like

Can you add an example of your secret.yaml (without the secrets of course)? did you added it manually to the cluster or you’ve put the YAML inside templates folder under the helm chart?

Hi @lekspek

Sure!
I use Argo CD to deploy Loki using its Helm chart and the configuration mapping the secrets as environment variables is defined in the values section for the helm chart. The secret file is located in another directory, that argocd reads when deploying/updating Loki.

The secret file looks something like:

apiVersion: v1
kind: Secret
metadata:
  name: <loki-secrets-name>
  namespace: <loki-namespace>
type: Opaque
data:
  grafana-loki-s3-accessKeyId: <base64-hashed string>
  grafana-loki-s3-secretAccessKey: <base64-hashed string>
  grafana-loki-s3-endpoint: <base64-hashed string>

So basically you didn’t use Pre-defined variables like described here: Grafana Loki configuration parameters | Grafana Loki documentation
You’ve just created a new environment variable and pointed the global s3 configuration to it, right?

As I use the Loki Helm chart I also use its values.yaml file.

  storage:
    ...
    s3:
      endpoint: "${GRAFANA-LOKI-S3-ENDPOINT}"
      accessKeyId: "${GRAFANA-LOKI-S3-ACCESKEYID}"
      secretAccessKey: "${GRAFANA-LOKI-S3-SECRETACCESSKEY}"
      ...

Where ‘endpoint’, ‘accessKeyId’ and ‘secretAccessKey’ are variables used by the helm chart. When processed results in the end in something like:

storage_config:
    endpoint: ${endpoint}
    access_key_id: ${accessKeyId}
    secret_access_key: ${secretAccessKey}
    ...

Yes, it can be quite confusing :slight_smile:

Hi all,

I have followed these steps and I am using sealed secrets to deploy the secret before deploying loki.
The sealed secrets controller creates the secret but then the grafana-agent-operator takes ownership of the secret and wipes the data values. Is there a setting somewhere to stop the grafana-agent-operator taking ownership of the secret.

apiVersion: v1
kind: Secret
metadata:
  creationTimestamp: "2024-03-01T09:30:22Z"
  labels:
    app.kubernetes.io/managed-by: grafana-agent-operator
  name: loki-secrets
  namespace: logging
  ownerReferences:
  - apiVersion: bitnami.com/v1alpha1
    controller: true
    kind: SealedSecret
    name: loki-secrets
    uid: 416ec23a-d50b-4470-aed6-fa5870f6a087
  - apiVersion: monitoring.grafana.com/v1alpha1
    blockOwnerDeletion: true
    kind: GrafanaAgent
    name: loki
    uid: ce95503e-7350-4717-905a-0907a918f413
  resourceVersion: "479732"
  uid: ceeea6ab-ec5d-4d53-81e8-01abd2c36e5e
type: Opaque

This can help too

type: s3
s3:
  endpoint: s3.eu-west-2.amazonaws.com
  region: eu-west-2
  accessKeyId: ${LOKI_AWS_ACCESS_KEY_ID}
  secretAccessKey: ${LOKI_AWS_SECRET_ACCESS_KEY}

kubectl create secret generic loki-aws-secrets --namespace Namespacename --from-literal=LOKI_AWS_ACCESS_KEY_ID= --from-literal=LOKI_AWS_SECRET_ACCESS_KEY=

the suggested solution does not seem to work, still getting a config error about closing braces. hard coding the access/secret key works, so something about parsing the env variable isn’t working

It seems that the error with ‘failed parsing config: missing closing brace’ message is occured when there are dashes in the name of the env variable.
So this config is not working for me

    s3:
      endpoint: "${GRAFANA-LOKI-S3-ENDPOINT}"
      accessKeyId: "${GRAFANA-LOKI-S3-ACCESKEYID}"
      secretAccessKey: "${GRAFANA-LOKI-S3-SECRETACCESSKEY}"

and this is working

    s3:
      endpoint: "${GRAFANALOKIS3ENDPOINT}"
      accessKeyId: "${GRAFANALOKIS3ACCESKEYID}"
      secretAccessKey: "${GRAFANALOKIS3SECRETACCESSKEY}"

Since you didn’t use a permalink, your link is broken.

I’m guessing your referred to loki/production/helm/loki/values.yaml at 7cd4890fdac7afe0a156f49b114bad0bea0bb7a7 · grafana/loki · GitHub maybe?

1 Like