Hi Grafana team,
I’m running Grafana Tempo on 3 EC2 instances in monolithic mode (single binary per node). All nodes use memberlist for the ring and write blocks to S3.
Config highlights:
3 EC2s, each running distributor + ingester + querier + compactor
memberlist with all 3 private IPs
Each node has:
unique private IP
unique ingester.lifecycler.id
unique tokens_file_path
replication_factor: 3
Access via AWS NLB
Observed behavior:
/compactor/ring shows all 3 nodes ACTIVE with ~33% ownership each (expected)
/ingester/ring shows only one node, with 100% ownership, regardless of which node/NLB endpoint I hit
Questions:
Is /ingester/ring node-local in monolithic Tempo, unlike /compactor/ring?
Does the ingester ring show global state only in distributed (microservices) mode?
Can ingestion replication still be working even if /ingester/ring shows a single node?
What’s the recommended way to verify ingester replication in this setup?
Trying to confirm whether this is expected behavior or a misconfiguration.
Thanks!