Azure Storage configuration for Tempo Distributed

Grafana Tempo
1

Hi

I wonder if anybody could help me with a Tempo deployment issue. I am deploying Tempo in Distributed Mode on an Azure AKS and using an Azure Storage Account for backend storage. This document suggests that I should apply the following configuration to my Helm values file:

tempo:
  storage:
    trace:
      backend: azure
      azure:
        container_name: container-name
        storage_account_name: storage-account-name
        storage_account_key: ${STORAGE_ACCOUNT_ACCESS_KEY}

  extraArgs:
    config.expand-env: true
  extraEnv:
    - name: STORAGE_ACCOUNT_ACCESS_KEY
      valueFrom:
        secretKeyRef:
          name: secret-name
          key: STORAGE_ACCOUNT_ACCESS_KEY

This does not appear to be working for me. When I apply the configuration against the ‘tempo’ block, the values do not get merged and the ‘local’ backend is still used. If I apply the configuration to the storage object, then the ‘trace’ values get merged but the ‘extraArgs’ and ‘extraEnv’ seem to be ignored - so that I get this error:

Error:
tempo-distributor-5f4c44d68-blwpt distributor level=error ts=2023-12-15T00:55:00.943447404Z caller=main.go:121 msg="error running Tempo" err="failed to init module services: error initialising module: usage-report: failed to initialize usage report: getting storage container: illegal base64 data at input byte 0"

Can anybody point me in the right direction?

2

So, I now have a working solution for this. It seems as though the ‘config.expand-env’ variable needs to be formatted as follows:

extraArgs:
  - "-config.expand-env=true"

I also needed to apply the extraArgs and extraEnv to each of the following services:

  • distributor
  • compactor
  • ingester
  • querier
  • query-frontend

The config in my custom values yaml file looks like this:


storage:
  trace:
    backend: azure
    azure:
      container_name: tempo-traces
      storage_account_name: stgappgeneraluks
      storage_account_key: ${STORAGE_ACCOUNT_ACCESS_KEY}  
  

distributor:
  log_received_spans:
    enabled: true
  extraArgs:
  - "-config.expand-env=true"
  extraEnv:
  - name: STORAGE_ACCOUNT_ACCESS_KEY
    valueFrom:
      secretKeyRef:
        name: tempo-traces-stg-key
        key: tempo-traces-key  

compactor:
  extraArgs:
  - "-config.expand-env=true"
  extraEnv:
  - name: STORAGE_ACCOUNT_ACCESS_KEY
    valueFrom:
      secretKeyRef:
        name: tempo-traces-stg-key
        key: tempo-traces-key  

ingester:
  extraArgs:
  - "-config.expand-env=true"
  extraEnv:
  - name: STORAGE_ACCOUNT_ACCESS_KEY
    valueFrom:
      secretKeyRef:
        name: tempo-traces-stg-key
        key: tempo-traces-key

querier:
  extraArgs:
  - "-config.expand-env=true"
  extraEnv:
  - name: STORAGE_ACCOUNT_ACCESS_KEY
    valueFrom:
      secretKeyRef:
        name: tempo-traces-stg-key
        key: tempo-traces-key
  

queryFrontend:
  extraArgs:
  - "-config.expand-env=true"
  extraEnv:
  - name: STORAGE_ACCOUNT_ACCESS_KEY
    valueFrom:
      secretKeyRef:
        name: tempo-traces-stg-key
        key: tempo-traces-key

That worked for me - although there is probabaly a more elegant solution. If not then maybe the documentation needs to be updated as perhaps the configuration values for the monolith and distributed versions of Tempo may well be different