Enable Local Storage for Grafana Tempo monolithic service on OpenShift

Grafana Tempo
1

Hi Team,

I want to enable local storage for the Grafana Tempo monolithic service on OpenShift

I could see examples for AWS S3, Azure, but I could not find a sample config for local storage.

I tried to configure the local storage using this config ( tempo/example/docker-compose/shared/tempo.yaml at 282775b007c7fd261d1be10a601ebe52161536c2 · grafana/tempo · GitHub ), then I am getting below error after starting the service
failed to init module services: error initialising module: store: failed to create store : unknown backend

I even tried to use PVC as local storage, but nothing seems to work.

Here are a few additional details:

tempo version - v2.9.0
branch - main
revision - dcbd3bdd7

@surajsidh Can you please help me with the setup?

Thanks,
Vishnu.

2

Could you please share your configs?

3

Hello, below is the configuration file (tempo.yaml) we are using for the Tempo monolithic service and trying to configure local as backend storage, but getting an error, which I mentioned in my previous message.

server:
  http_listen_port: 3200

query_frontend:
  search:
    duration_slo: 5s
    throughput_bytes_slo: 1.073741824e+09
  trace_by_id:
    duration_slo: 5s

distributor:
  log_received_spans:
    enabled: true
  receivers:
    otlp:
      protocols:
        http:
        grpc:

ingester:
  max_block_duration: 5m

compactor:
  compaction:
    block_retention: 24h

metrics_generator:
  processor:
    span_metrics:
      dimensions:
        - operation_name
        - graphql_query
  storage: 
    path: /tmp/tempo/generator/wal
    remote_write:
      - url: http://prometheus:9090/api/v1/write
        send_exemplars: true

storage:
  trace:
    backend: local
    local:
      path: /var/tempo/blocks
    wal:
      path: /var/tempo/wal

overrides:
  defaults:
    metrics_generator:
      processors: [service-graphs, span-metrics]
4

I’ve provided the config file as requested. If you are up for a quick call to debug the issue, please feel free to email me at vishnu132.d@gmail.com.

5

From source code, unknown backend error should also provide what backend was configured. I copied the config you provided to example docker compose and could not reproduce the problem. However, if I put backend: '', an empty content to the yaml file or without -config.file=/etc/tempo.yaml I do see the same error: failed to init module services: error initialising module: store: failed to create store: unknown backend (with empty space at the end).

I would double-check if you do have the right config and provide it in command arguments (tempo/example/docker-compose/local/docker-compose.yaml at 282775b007c7fd261d1be10a601ebe52161536c2 · grafana/tempo · GitHub).

6

Hello, as per your suggestion, I added the -config.file=/etc/tempo.yaml flag to the command arguments, and that resolved the initial issue. However, I’m now seeing that the “memberlist-kv” module failed to start, which in turn triggered dependency module errors.

The specific error message is:

msg="module failed" module=memberlist-kv err"=starting module memberlist-kv: invalid service state: Stopping, expected: Running"

Can you please let me know what is causing the memberlist-kv module to fail?

Thanks.

7

Hello @yaffle , Can you please let me know why the memberlist-kv module failes to initialize.