tempo version 2.8.2
deployment mode microservice
platform kubernetes using helm chart
I’m looking for guidance on setting up Grafana Tempo in a high-availability (HA) configuration.
Currently, I have two Kubernetes clusters — one in the data center 1 and another in the data center 2. Tempo (in microservices mode) is already deployed and working well in one cluster. I plan to deploy another Tempo instance in the second cluster for resiliency in case of a data center failure.
i can front it with LB to keep endpoints same for failover
What would be the best and most reliable approach for HA setup in this scenario?
→ two independent S3 storages and configure OpenTelemetry (OTEL) to export traces to both Tempo distributors but this doubles storage , processing compaction everything ?
→ Use a common S3 bucket and ensure only one compactor runs at a time, possibly by leveraging a shared key-value store or coordination mechanism between clusters. is this possible ?
in short can i improve this architecture to reduce the cost of dobule ingestion and storage
and also main problem here say one of the cluster is down for maintaince and then comes back up
now S3 will not be in sync
so i really want to know if common s3 setup can work and how to configure it
+-------------------+
| OTel Collector |
|-------------------|
| exporters: |
| 1. tempo-1 |
| 2. tempo-2 |
+---------+---------+
|
┌───────┴────────┐
▼ ▼
±---------------+ ±---------------+
| Tempo-DC 1 | | Tempo- DC 2 |
| (Distributor, | | (Distributor, |
| Ingester, | | Ingester, |
| Compactor, | | Compactor, |
| S3 bucket A) | | S3 bucket B) |
±---------------+ ±---------------+
▼ ▼
S3 bucket A S3 bucket B