The issue of tempo cluster with docker-compose and host network

I’m concerned about deploying a Tempo cluster on two host machines (52 and 151) using Docker Compose and network hosting. The Tempo.yaml file is as follows:

server:
  http_listen_port: 3200
  grpc_listen_port: 9096  

distributor:
  ring:
    kvstore:
      store: memberlist
    instance_addr: ${ip_52}
    instance_id: "tempo-52"
  receivers:
    otlp:
      protocols:
        http:
          endpoint: 0.0.0.0:43180
        grpc:
          endpoint: 0.0.0.0:43170

ingester:
  max_block_duration: 15m
  lifecycler:
    ring:
      kvstore:
        store: memberlist
      replication_factor: 2
    address: ${ip_52}
    id: tempo-52

compactor:
  compaction:
    block_retention: 24h
    compacted_block_retention: 1h
  ring:
    kvstore:
      store: memberlist
    instance_addr: ${ip_52}
    instance_id: "tempo-52"

storage:
  trace:
    blocklist_poll: 15m
    blocklist_poll_tenant_index_builders: 1
    backend: s3
    block:
      bloom_filter_false_positive: .05
    wal:
      path: /var/tempo/wal
    s3:
      bucket: ${bucket}
      endpoint: s3.amazonaws.com
      region: ${region}
      access_key: ${access_key}
      secret_key: ${secret_key}
      prefix: observer/
      insecure: true
    local:
      path: /var/tempo/traces
    pool:
      max_workers: 100
      queue_depth: 10000

overrides:
  defaults:
    metrics_generator:
      processors: ["service-graphs", "span-metrics", "local-blocks"]
      generate_native_histograms: both

metrics_generator:
  registry:
    external_labels:
      source: tempo
      cluster: distributed
  storage:
    path: /var/tempo/generator/wal
    remote_write:
      - url: http://${mimir-host}/api/v1/push
        send_exemplars: true
  traces_storage:
    path: /var/tempo/generator/traces

memberlist:
  node_name: "tempo-52"
  join_members:
    - ${ip_52}:7949
    - ${ip_151}:7949
  bind_port: 7949
  bind_addr: 
    - ${ip_52}
  advertise_addr: ${ip_52}
  advertise_port: 7949
  gossip_interval: 500ms

The tempo.yaml file on machine 151 is similar to the one described above, only the IP addresses are different.

After deployment, I found that on ingester/ring, I can only see various data, but no cluster data is displayed.

memberlist:

When I send a trace to 52 tempo, I can only find the corresponding trace from 52 tempo, but not from 151.
I want to achieve the effect of a cluster. Can anyone help me see what the problem is?

1 Like