Tempo multi-region architecture - query architecture

Hi,

I’m looking to deploy Tempo in multiple regions while having a centralized Grafana instance. The least complex approach I can think of would be to have a separate Tempo deployment in each region, but that would require that I expose the query frontend in each region which I’d rather avoid. Given that Tempo uses object storage, I was wondering if I could deploy the ingester/compactors in each region, and then colocate the querier(s) with Grafana.

  • Is there a viable approach to have ingesters/compactors and queriers collaborate with each other without direct communication? Based on the documentation I’m guessing that this would require a central gossip server, but if there’s a simpler solution I’d be keen to hear it.
  • If there is a solution for the above, is there any significant issue with multiple ingester deployments sharing the same querier?
  • If the above is not possible, would I instead be able to configure two Tempo deployments (the ingester one and the querier one) to have independently consistent hash rings? I’m thinking something along the lines of both clusters using the same RNG seed.