[Loki] Helm Chart cannot parse environment variables with "- config.extra-env=true"

Grafana Loki
1

Hey guys,

I am struggling with getting environment variables to work.
Loki Helm Chart cannot parse environment variables. I am using FluxV2 and when I pass - config.extra-env=true in extraArgs in the HelmRelease, the following error is produced:

loki failed parsing config: /etc/loki/config/config.yaml: yaml: control characters are not allowed

The environment variables are used for s3 storage. So in backend I am enabling - config.extra-env=true in extraArgs and then env in extraEnv.

loki-helmrelease.yaml

apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: loki
  namespace: monitoring
spec:
  interval: 24h
  upgrade:
    crds: CreateReplace
  chart:
    spec:
      chart: loki
      version: 5.43.5
      sourceRef:
        kind: HelmRepository
        name: loki
  values:
    global:
      image:
        # -- Overrides the Docker registry globally for all images
        registry: null
    loki:
      # Configures the readiness probe for all of the Loki pods
      readinessProbe:
        httpGet:
          path: /ready
          port: http-metrics
        initialDelaySeconds: 30
        timeoutSeconds: 1
      image:
        # -- The Docker registry
        registry: docker.io
        # -- Docker image repository
        repository: grafana/loki
      # -- Provides a reloadable runtime configuration file for some specific configuration
      # -- Storage config. Providing this will automatically populate all necessary storage configs in the templated config.
      storage:
        bucketNames:
          chunks:  test-loki
          ruler:  test-loki
          admin: test-loki
        type: s3
        s3:
          s3: "s3://test-loki"
          endpoint: "test.se-grid.com/"
          region: eu-north-1
          secretAccessKey: "${SECRET_KEY}"
          accessKeyId: ${ACCESS_KEY}
          signatureVersion: null
          s3ForcePathStyle: true
          insecure: false
          http_config: {}
    
    # Configuration for the backend pod(s)
    backend:
      # -- Number of replicas for the backend
      replicas: 3
      autoscaling:
        # -- Enable autoscaling for the backend.
        enabled: false
      # -- Comma-separated list of Loki modules to load for the read
      targetModule: "backend"
      # -- Additional CLI args for the backend
      #! Enable expand-env so we can add environment variable for storagegrid secrets
      extraArgs:
        - "-config.expand-env=true"
      extraEnv:
        - name: SECRET_KEY
          valueFrom:
            secretKeyRef:
              name: loki-test-credentials
              key: secretAccessKey
        - name: ACCESS_KEY
          valueFrom:
            secretKeyRef:
              name: loki-test-credentials
              key: accessKeyId
      # -- Volume mounts to add to the backend pods
      extraVolumeMounts: []
      # -- Volumes to add to the backend pods
      extraVolumes: []
      # -- Resource requests and limits for the backend
      resources: {}

After I push the changes I get the above error yaml: control characters are not allowed

I am using:

  • Loki Helm chart version: 5.43.5
  • Infrastructure: Kubernetes
  • Deployment tool: Fluxv2

I also tried to render it and it is not rendered correctly. What am I missing? I have also created a bug in grafana/loki GitHub

Thank you!

/Angelos

2

I mostly solved the problem. I had to use extraEnvFrom instead of extraEnv so it could use the secret that I had created. #12218

However, I am facing another error now showing “NoCredentialsProviders”:

loki level=error ts=2024-03-18T08:12:12.75711169Z caller=table.go:342 table-name=loki_index_19800 org_id=fake msg="index set has some problem, cleaning it up" err="NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors

3

I solved the issue. See the solution in Github:
[Loki] Helm Chart cannot parse environment variables with "- config.extra-env=true" · Issue #12218 · grafana/loki · GitHub