# Provide S3 credentials using environment variables

**URL:** <https://community.grafana.com/t/provide-s3-credentials-using-environment-variables/100132>\
**Category:** Grafana Loki\
**Tags:** helm, loki\
**Created:** [July 31, 2023, 8:36am UTC](https://community.grafana.com/t/provide-s3-credentials-using-environment-variables/100132 "2023-07-31T08:36:43Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![lekspek](https://sea2.discourse-cdn.com/grafana/user_avatar/community.grafana.com/lekspek/32/36724_2.png) [@lekspek](https://community.grafana.com/u/lekspek)\
**Post date:** [July 31, 2023, 8:36am UTC](https://community.grafana.com/t/provide-s3-credentials-using-environment-variables/100132/1 "2023-07-31T08:36:43Z")

</div>

Is there any option to provide Loki S3 credentials using environment variables? I’m using Hashicorp vault and thinking about how to securely pass keys, but seems no option is secure enough.  
We don’t use EKS or any cloud, everything is on-prem.

---

<div class="post-metadata">

**Author:** ![b0b](https://sea2.discourse-cdn.com/grafana/user_avatar/community.grafana.com/b0b/32/8761_2.png) [@b0b](https://community.grafana.com/u/b0b)\
**Post date:** [July 31, 2023, 1:18pm UTC](https://community.grafana.com/t/provide-s3-credentials-using-environment-variables/100132/2 "2023-07-31T13:18:12Z")

</div>

Take a look at [Use environment variables in the configuration](https://grafana.com/docs/loki/latest/configuration/#use-environment-variables-in-the-configuration) in the Loki docs.

Are you running Kubernetes on-prem? If not, what configuration management system do you use?

EDIT

As you have tagged your post with Helm, I assume you use something Kubernetes-like…

This might help [https://community.grafana.com/t/storing-s3-accesskeyid-and-secretaccesskey-securely](https://community.grafana.com/t/storing-s3-accesskeyid-and-secretaccesskey-securely)

/EDIT

---

<div class="post-metadata">

**Author:** ![lekspek](https://sea2.discourse-cdn.com/grafana/user_avatar/community.grafana.com/lekspek/32/36724_2.png) [@lekspek](https://community.grafana.com/u/lekspek)\
**Post date:** [August 2, 2023, 11:18am UTC](https://community.grafana.com/t/provide-s3-credentials-using-environment-variables/100132/3 "2023-08-02T11:18:57Z")

</div>

Yes I’m running on-prem. what information is missing here?

I’ve followed the link you sent and tried to improve my Yaml:

```auto
loki:
  loki:
    enabled: true
    auth_enabled: false
    storage:
      bucketNames:
        chunks: loki
        ruler: loki
        admin: loki
      type: s3
      s3:
        accessKeyId: "${GRAFANA-LOKI-S3-ACCESKEYID}"
        secretAccessKey: "${GRAFANA-LOKI-S3-SECRETACCESSKEY}"
        region: eu-west-1
    commonConfig:
      replication_factor: 1
    limits_config:
      retention_period: 48h
      retention_stream:
        - selector: '{namespace="monitoring"}'
          priority: 1
          period: 24h
        - selector: '{namespace="loki"}'
          priority: 2
          period: 24h
  ingress:
    enabled: true
    ingressClassName: "nginx"
    paths:
      write:
        - /api/prom/push
        - /loki/api/v1/push
      read:
        - /api/prom/tail
        - /loki/api/v1/tail
        - /loki/api
        - /api/prom/rules
        - /loki/api/v1/rules
        - /prometheus/api/v1/rules
        - /prometheus/api/v1/alerts
      singleBinary:
        - /api/prom/push
        - /loki/api/v1/push
        - /api/prom/tail
        - /loki/api/v1/tail
        - /loki/api
        - /api/prom/rules
        - /loki/api/v1/rules
        - /prometheus/api/v1/rules
        - /prometheus/api/v1/alerts
    hosts:
      - loki.domain.local
  write:
    replicas: 1
    persistence:
      storageClass: "netapp-storage"
  read:
    replicas: 1
    persistence:
      storageClass: "netapp-storage"
  backend:
    persistence:
      storageClass: "netapp-storage"
    extraArgs:
      - '-config.expand-env=true'
    extraEnv:
      - name: GRAFANA-LOKI-S3-ACCESKEYID
        valueFrom:
          secretKeyRef:
            name: loki-secrets
            key: grafana-loki-s3-accessKeyId
      - name: GRAFANA-LOKI-S3-SECRETACCESSKEY
        valueFrom:
          secretKeyRef:
            name: loki-secrets
            key: grafana-loki-s3-secretAccessKey
  singleBinary:
    persistence:
      storageClass: "netapp-storage"
    replicas: 1

```

and here is example of my secret:

```auto
apiVersion: v1
kind: Secret
type: Opaque
metadata:
  name: loki-secrets
data:
  grafana-loki-s3-accessKeyId: =
  grafana-loki-s3-secretAccessKey: ==

```

(censored my keys)  
but after the helm deployment, the keys are added and removed saying:

> Controlled By GrafanaAgent loki

example: [https://i.imgur.com/tikfFe2.png](https://i.imgur.com/tikfFe2.png)

Edit:  
Seem after `helm upgrade` it’s forcing the secret to be updated and re-apply it. but now **backend** container restart over and over… with this log:

```auto
failed parsing config: missing closing brace

```

Edit2:  
When I’m trying to create the secret and then install Loki I get this error:

```auto
Error: INSTALLATION FAILED: rendered manifests contain a resource that already exists. Unable to continue with install: Secret "loki-secrets" in namespace "loki" exists and cannot be imported into the current release: invalid ownership metadata; label validation error: missing key "app.kubernetes.io/managed-by": must be set to "Helm"; annotation validation error: missing key "meta.helm.sh/release-name": must be set to "loki"; annotation validation error: missing key "meta.helm.sh/release-namespace": must be set to "loki"

```

---

<div class="post-metadata">

**Author:** ![lekspek](https://sea2.discourse-cdn.com/grafana/user_avatar/community.grafana.com/lekspek/32/36724_2.png) [@lekspek](https://community.grafana.com/u/lekspek)\
**Post date:** [August 6, 2023, 10:57am UTC](https://community.grafana.com/t/provide-s3-credentials-using-environment-variables/100132/4 "2023-08-06T10:57:14Z")

</div>

Assume I’ve installed Hashicorp Vault server. Now I can map the keys inside the pod, what will be the best practice to load them?  
I’ve tried to add this:

```auto
  singleBinary:
    persistence:
      storageClass: "netapp-storage"
    replicas: 1
    extraArgs:
      - ' &&'
      - 'source /vault/secrets/observability-loki-s3aws'
    podAnnotations:
      vault.hashicorp.com/agent-inject: 'true'
      vault.hashicorp.com/role: 'loki'
      vault.hashicorp.com/agent-inject-secret-observability-loki-s3aws: 'internal/data/observability/loki/s3aws'
      vault.hashicorp.com/agent-inject-template-observability-loki-s3aws: |
        {{ with secret "internal/data/observability/loki/s3aws" -}}
          export AWS_STORAGE_ACCESS_KEY="{{ .Data.data.access_key_id }}" &&
          export AWS_STORAGE_SECRET_KEY="{{ .Data.data.secret_access_key }}" &&
          echo "Done!!" ~~~
        {{- end }}

```

But Loki don’t use this script when pod is starting.

@b0b hope you can help here.

---

<div class="post-metadata">

**Author:** ![b0b](https://sea2.discourse-cdn.com/grafana/user_avatar/community.grafana.com/b0b/32/8761_2.png) [@b0b](https://community.grafana.com/u/b0b)\
**Post date:** [August 7, 2023, 7:42am UTC](https://community.grafana.com/t/provide-s3-credentials-using-environment-variables/100132/5 "2023-08-07T07:42:38Z")

</div>

I can make some educated guesses 🙂

We are not using the Vault Agent Injector so I have no personal experience with it but looks like the secrets are put in a file under `/vault/secrets/`.

If the rendered secrets file would be in a format the AWS SDK can use (more [info](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html)) then you could configure the pod with ´AWS\_CONFIG\_FILE`=\<path\_to\_secrets\_file\>.

I think you would then leave out all the AWS config from the Loki config, so no `grafana-loki-s3-accessKeyId` or `grafana-loki-s3-secretAccessKey` or anything like that. AWS SDK, which Loki uses for S3 connections, should take care of the S3 auth using the config file set by `AWS_CONFIG_FILE`.

Something like that. We no longer use `AWS_STORAGE_ACCESS_KEY` or `AWS_STORAGE_SECRET_KEY` in our Loki config. Instead we use service accounts. [This](https://epam.github.io/edp-install/operator-guide/loki-irsa/) looks like what we are doing.

Anyway, there are many ways to do this. You can definitely do it like you are attempting to do it. I just don’t know exactly how to configure it as I’m not using the same method.

---

<div class="post-metadata">

**Author:** ![lekspek](https://sea2.discourse-cdn.com/grafana/user_avatar/community.grafana.com/lekspek/32/36724_2.png) [@lekspek](https://community.grafana.com/u/lekspek)\
**Post date:** [August 7, 2023, 8:59am UTC](https://community.grafana.com/t/provide-s3-credentials-using-environment-variables/100132/6 "2023-08-07T08:59:35Z")

</div>

So from what you’re saying - currently you support only IAM via EKS (based on the profile you provided - the service account is the way of EKS to manage permissions between services).  
From what you saying about `AWS_CONFIG_FILE`, if file is provided on a specific path with AWS format, then provide `AWS_CONFIG_FILE` the path to the file, and AWS SDK will use it?  
If so it can be a great solution. (but better to support native secret)

---

<div class="post-metadata">

**Author:** ![b0b](https://sea2.discourse-cdn.com/grafana/user_avatar/community.grafana.com/b0b/32/8761_2.png) [@b0b](https://community.grafana.com/u/b0b)\
**Post date:** [August 7, 2023, 9:22am UTC](https://community.grafana.com/t/provide-s3-credentials-using-environment-variables/100132/7 "2023-08-07T09:22:31Z")

</div>

> [@lekspek](#):
>
> From what you saying about `AWS_CONFIG_FILE`, if file is provided on a specific path with AWS format, then provide `AWS_CONFIG_FILE` the path to the file, and AWS SDK will use it?

That is what I would expect to happen. I have not tried it.

---

<div class="post-metadata">

**Author:** ![lekspek](https://sea2.discourse-cdn.com/grafana/user_avatar/community.grafana.com/lekspek/32/36724_2.png) [@lekspek](https://community.grafana.com/u/lekspek)\
**Post date:** [August 7, 2023, 2:44pm UTC](https://community.grafana.com/t/provide-s3-credentials-using-environment-variables/100132/8 "2023-08-07T14:44:59Z")

</div>

> [@b0b](#):
>
> AWS\_CONFIG\_FILE

So I made it work, Loki & Promtail chart using credentials stored in HashiCorp Vault.  
Here’s my `values.yaml` in case it can help someone: (single binary is comment since we don’t need it right now)

```auto
loki:
  loki:
    enabled: true
    auth_enabled: false
    storage:
      bucketNames:
        chunks: loki
        ruler: loki
        admin: loki
      type: s3
      s3:
        region: eu-west-1
    commonConfig:
      replication_factor: 1
    limits_config:
      retention_period: 48h
      retention_stream:
        - selector: '{namespace="monitoring"}'
          priority: 1
          period: 24h
        - selector: '{namespace="loki"}'
          priority: 2
          period: 24h
  ingress:
    enabled: true
    ingressClassName: "nginx"
    paths:
      write:
        - /api/prom/push
        - /loki/api/v1/push
      read:
        - /api/prom/tail
        - /loki/api/v1/tail
        - /loki/api
        - /api/prom/rules
        - /loki/api/v1/rules
        - /prometheus/api/v1/rules
        - /prometheus/api/v1/alerts
      singleBinary:
        - /api/prom/push
        - /loki/api/v1/push
        - /api/prom/tail
        - /loki/api/v1/tail
        - /loki/api
        - /api/prom/rules
        - /loki/api/v1/rules
        - /prometheus/api/v1/rules
        - /prometheus/api/v1/alerts
    hosts:
      - loki.dc-infra.local
  write:
    replicas: 1
    persistence:
      storageClass: "netapp-storage"
    extraArgs:
      - '-config.expand-env=true'
    podAnnotations:
      vault.hashicorp.com/agent-inject: 'true'
      vault.hashicorp.com/role: 'loki'
      vault.hashicorp.com/agent-inject-secret-observability-loki-s3aws: 'internal/data/observability/loki/s3aws'
      vault.hashicorp.com/agent-inject-template-observability-loki-s3aws: |
        {{ with secret "internal/data/observability/loki/s3aws" -}}
        [default]
        aws_access_key_id={{ .Data.data.access_key_id }}
        aws_secret_access_key={{ .Data.data.secret_access_key }}
        {{- end }}
    extraEnv:
      - name: AWS_SHARED_CREDENTIALS_FILE
        value: "/vault/secrets/observability-loki-s3aws"
  read:
    replicas: 1
    persistence:
      storageClass: "netapp-storage"
    extraArgs:
      - '-config.expand-env=true'
    podAnnotations:
      vault.hashicorp.com/agent-inject: 'true'
      vault.hashicorp.com/role: 'loki'
      vault.hashicorp.com/agent-inject-secret-observability-loki-s3aws: 'internal/data/observability/loki/s3aws'
      vault.hashicorp.com/agent-inject-template-observability-loki-s3aws: |
        {{ with secret "internal/data/observability/loki/s3aws" -}}
        [default]
        aws_access_key_id={{ .Data.data.access_key_id }}
        aws_secret_access_key={{ .Data.data.secret_access_key }}
        {{- end }}
    extraEnv:
      - name: AWS_SHARED_CREDENTIALS_FILE
        value: "/vault/secrets/observability-loki-s3aws"
  backend:
    replicas: 1
    persistence:
      storageClass: "netapp-storage"
    extraArgs:
      - '-config.expand-env=true'
    podAnnotations:
      vault.hashicorp.com/agent-inject: 'true'
      vault.hashicorp.com/role: 'loki'
      vault.hashicorp.com/agent-inject-secret-observability-loki-s3aws: 'internal/data/observability/loki/s3aws'
      vault.hashicorp.com/agent-inject-template-observability-loki-s3aws: |
        {{ with secret "internal/data/observability/loki/s3aws" -}}
        [default]
        aws_access_key_id={{ .Data.data.access_key_id }}
        aws_secret_access_key={{ .Data.data.secret_access_key }}
        {{- end }}
    extraEnv:
      - name: AWS_SHARED_CREDENTIALS_FILE
        value: "/vault/secrets/observability-loki-s3aws"
# singleBinary:
# persistence:
# storageClass: "netapp-storage"
# replicas: 0
# extraArgs:
# - '-config.expand-env=true'
# podAnnotations:
# vault.hashicorp.com/agent-inject: 'true'
# vault.hashicorp.com/role: 'loki'
# vault.hashicorp.com/agent-inject-secret-observability-loki-s3aws: 'internal/data/observability/loki/s3aws'
# vault.hashicorp.com/agent-inject-template-observability-loki-s3aws: |
# {{ with secret "internal/data/observability/loki/s3aws" -}}
# [default]
# aws_access_key_id={{ .Data.data.access_key_id }}
# aws_secret_access_key={{ .Data.data.secret_access_key }}
# {{- end }}
# extraEnv:
# - name: AWS_SHARED_CREDENTIALS_FILE
# value: "/vault/secrets/observability-loki-s3aws"

promtail:
  enabled: false
  config:
    logLevel: info
    clients:
      - url: http://loki-gateway.loki.svc.cluster.local/loki/api/v1/push

```

For a better reference here is the dependencies part from my `Chart.yaml`:

```auto
dependencies:
  - name: loki
    condition: loki.enabled
    repository: https://grafana.github.io/helm-charts
    version: 5.10.0
  - name: promtail
    condition: promtail.enabled
    repository: https://grafana.github.io/helm-charts
    version: 6.14.1

```

Thanks for the resources.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/grafana/original/2X/e/ef5308cc58c9e5a13187b5012eb99f406928462f.png) [@system](https://community.grafana.com/u/system)\
**Post date:** [August 6, 2024, 2:45pm UTC](https://community.grafana.com/t/provide-s3-credentials-using-environment-variables/100132/9 "2024-08-06T14:45:26Z")

</div>

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.