I am trying to run tempo-distributed in kubernetes but getting the following error messages
"Failed to resolve tempo-distributed-gossip-ring: lookup tempo-distributed-gossip-ring on 10.96.0.10:53: no such host" │ "Failed to resolve tempo-distributed-gossip-ring: lookup tempo-distributed-gossip-ring on 10.96.0.10:53: server misbehaving" msg="pusher failed to consume trace data" err="DoBatch: InstancesCount <= 0"
I have been reading and the reason is because the distributed and ingester can’t communicate. They need to be in the same gossip ring. There is a an issue related to this in this forum but they are using etcd for KV but it is not my case.
Is this gossip ring new to tempo? I don’t remember having this issue in the past.
The question is:
How can I create a gossip ring for tempo? Is there an article showing the steps because I am at lost?
My config file looks like this:
multitenancy_enabled: false compactor: compaction: block_retention: 48h ring: kvstore: store: memberlist distributor: ring: kvstore: store: memberlist receivers: zipkin: endpoint: 0.0.0.0:9411 querier: frontend_worker: frontend_address: tempo-distributed-query-frontend-discovery:9095 ingester: lifecycler: ring: replication_factor: 1 kvstore: store: memberlist tokens_file_path: /var/tempo/tokens.json memberlist: abort_if_cluster_join_fails: false join_members: - tempo-distributed-gossip-ring overrides: per_tenant_override_config: /conf/overrides.yaml server: http_listen_port: 3100 storage: trace: azure: .....