Enable Local Storage for Grafana Tempo monolithic service in Linux

Grafana Tempo
1

Hi Team,

I want to enable local storage for Grafana Tempo monolithic service in Linux

I could see examples for AWS S3, Azure but I could not find sample config for local storage

Kindly help me to prepare the configuration file

Regards,
Gopi

2

checkout configuration page in our docs for details on ways to configure tempo and tempo/example at main · grafana/tempo · GitHub for examples of configs.

3

Hi

Thanks for the response.

I could not find the config file in Git repo under examples folder, to understand how local storage configuration is enabled

I tried to configure the local storage then I am getting below error after starting the service

failed to init module services error initialising module: store: failed to create store mkdir : no such file or directory

4

hi, here is one example where local backend is configured to store data locally, can you try this example config, and see if it works.

also, can you make sure the configured directory exists on the disk, and it has correct permissions for tempo to read and write data in that directory.

failed to init module services error initialising module: store: failed to create store mkdir : no such file or directory makes me think that either it can’t find configured dir.

if this doesn’t work, can you share more details like the OS, filesystem, tempo version, and the tempo config you are using.

5

Hi surajsidh ,

I’m trying to run grafana/tempo service through docker compose and also trying to use local storage for storing traces and faced the same error -

msg=“error running Tempo” err="failed to init module services: error initialising module: store: failed to create store: unknown backend ",

Here i’m sharing the installation yaml file and config file for troubleshooting. So, kindly reply back what the mistake i’ve made.

docker-compose.yaml–>

version: '3'

services:
  tempo:
    image: grafana/tempo:latest
    ports:
      - "3200:3200"
      - "4317:4317"
      - "4318:4318"
    command: ./tempo-config.yaml=/etc/tempo/tempo-config.yaml
    restart: unless-stopped

tempo-config.yaml–>

  http_listen_port: 3200

distributor:
  receivers:                           # this configuration will listen on all ports and protocols that tempo is capable of.
    jaeger:                            # the receives all come from the OpenTelemetry collector.  more configuration information can
      protocols:                       # be found there: https://github.com/open-telemetry/opentelemetry-collector/tree/main/receiver
        thrift_http:                   #
        grpc:                          # for a production deployment you should only enable the receivers you need!
        thrift_binary:
        thrift_compact:
    zipkin:
    otlp:
      protocols:
        http:
        grpc:
    opencensus:

ingester:
  max_block_duration: 5m               # cut the headblock when this much time passes. this is being set for demo purposes and should probably be left alone normally

compactor:
  compaction:
    block_retention: 1h                # overall Tempo trace retention. set for demo purposes

metrics_generator:
  registry:
    external_labels:
      source: tempo
      cluster: docker-compose
  storage:
    path: /tmp/tempo/generator/wal
    remote_write:
      - url: http://prometheus:9090/api/v1/write
        send_exemplars: true

storage:
  trace:
    backend: local                     # backend configuration to use
    wal:
      path: /tmp/tempo/wal             # where to store the the wal locally
    local:
      path: /tmp/tempo/blocks

overrides:
  metrics_generator_processors: [service-graphs, span-metrics] # enables metrics generator
type or paste code here

When i run the docker compose file, the container was in restarting state.Also created directories as given in tempo-config.yaml file, expecting rply back as soon as possible from you.

Regards,
Prasand.M

6

the config block looks correct to me, can you check:

  1. if yaml is indented correctly
  2. docker process has permissions to access these paths.
  3. docker-compose is loading tempo-config.yaml correctly.

here is a working tempo.yaml for reference, and here is a complete docker-compose setup for running tempo locally…