Metrics Generator needs blob storage config?

Grafana Tempo
1

I’m currently using Azure Blob Storage as a backend for Tempo, as well as the Bitnami chart for Tempo. When I pass -config.expand-env=true for all 5 components as per the official configuration, all of the components seem to run fine, except for the metrics-generator pod, which keeps on crash loop back offing.

It crashes with this error message:

│ level=error ts=2025-02-12T00:29:55.338979444Z caller=main.go:124 msg="error running Tempo" err="failed to init module services: error initialising module: optional-store: failed to create store:  │
│ getting storage container: decode account key: illegal base64 data at input byte 0"                                                                                                                 │

When I pass the flag “-config.expand-env=true” and the secret, it seems to be healthy again. My question is–why? As far as I can tell by reading through the metrics-generator configuration, blob storage is not needed at any point. So why would the metrics-generator fail to run without the additional flag?

This is the configuration for the backend:

      storage:
        trace:
          backend: azure
          azure:
            container_name: tempo
            storage_account_name: storage01
            storage_account_key: ${STORAGE_ACCOUNT_ACCESS_KEY}
          wal:
            path: {{ .Values.tempo.dataDir }}/wal
2

Hi, blob storage is required if you enable the localblocks processor. This processor is needed for TraceQL metrics queries. It produces blocks that are stored in the backend.

It is briefly described here: Configure TraceQL metrics | Grafana Tempo documentation