We recently completed our migration to Grafana Mimir 3.0, and the results have been nothing short of remarkable. Real-world resource metrics showed significant improvements, and the performance gains were immediately noticeable across our infrastructure.
We’re running 9 mimir-ingesters, and the changes captured after migration were striking.
[Mimir] Ingester Memory — Before vs After Mimir 3.0 Migration
Per-pod average memory usage was previously hovering around 6–7 GiB. After migrating to Mimir 3.0, that average dropped dramatically to approximately 2 GiB.
New Architecture: Kafka Integration
So where did that load go? Mimir 3.0 introduced a fundamental architectural change.
As outlined in the official release blog:
The write path now follows a Distributor → Kafka → Ingester pipeline. A new mimir-kafka component has been introduced, offloading buffering responsibility from the ingesters.
[Mimir] Distributor Memory — Before vs After Mimir 3.0 Migration
Farewell, Consul — The Biggest Win
Previously, using the HA Tracker required either Consul or etcd as a KV store. In EKS environments, Consul was effectively the only practical option.
However, Consul comes with well-documented operational challenges:
Over approximately 3 years of operating Consul, we repeatedly encountered these issues and had to resolve them manually every time.
On top of that, if you don’t explicitly disable the Connect Injector during the initial Consul Helm installation, it gets deployed alongside Consul. When the injector’s memory usage spikes unexpectedly and triggers an OOMKilled event, the resulting pod panic state can prevent ReplicaSet controllers from reconciling the desired replica count — effectively freezing your scaling operations.
With Mimir 3.0, Consul is no longer required.
The KV store for HA deduplication now defaults to memberlist, eliminating the Consul dependency entirely. This single improvement alone makes the Mimir 3.0 upgrade worthwhile.
As fate would have it, we were hit with a Consul raft.db corruption issue right before our planned migration — which accelerated our timeline. The migration resolved the issue and delivered outstanding performance improvements in one stroke.
Cost Impact: Resource Reduction Estimate
The dramatic reduction in resource requirements directly translates to infrastructure cost savings.
Projected Cost Savings at Scale (100 vCPU Workload Basis)
To illustrate the cost impact at scale, we projected the observed reduction ratios (~46% CPU reduction, ~77% memory reduction) onto a 100 vCPU workload and mapped the requirements to AWS m7g instances (us-east-1, On-Demand pricing).
Scaling to 100 vCPU equivalent workload:
vCPU Required
Memory Required
Before (Mimir 2.x)
100 cores
~583 GiB
After (Mimir 3.0)
~54 cores
~129 GiB
Projected based on observed memory-per-core ratio before and after migration.
Note: Before migration, memory is the bottleneck (583 GiB needed vs. 4 GiB/vCPU ratio of m7g), requiring over-provisioning on CPU to meet memory demands.
Monthly & Annual Savings:
Monthly (730 hrs)
Annual
Before
$5,718.53
$68,622.34
After
$1,906.18
$22,874.11
Savings
$3,812.35/mo
$45,748.23/yr
Reduction
~66.7%
~66.7%
These figures are based on On-Demand pricing. With Reserved Instances or Savings Plans, the absolute savings would differ, but the relative improvement remains consistent.
Additionally, removing Consul from the infrastructure eliminates its own compute, memory, and operational overhead — which is not reflected in the numbers above but represents further savings.
Summary
Improvement
Details
Ingester Memory
6–7 GiB → 2 GiB per pod (~70% reduction)
Consul Dependency
Eliminated (memberlist default)
Architecture
New Kafka-based write path
Estimated Cost Savings
~$45K/year at 100 vCPU scale
Mimir 3.0 is a massive leap forward. The combination of architectural improvements, resource efficiency, and the removal of the Consul dependency makes this one of the most impactful upgrades we’ve done.
If you’re still on Mimir 2.x and have been dealing with Consul headaches or high ingester resource usage — don’t wait. The migration is well worth it.
These figures are based on On-Demand pricing. With Reserved Instances or Savings Plans, the absolute savings would differ, but the relative improvement remains consistent.