I am planning migration of Mimir OpenSource from v2.17 to 3.x with new Kafka based ingest storage architecture.
According to documentation, I need to run two Mimir deployments in parallel (one existing classic architecture and one with new Kafka based architecture) and duplicate data to both with the configuration updates in Alloy.
But this does not include migration of old telemetry from classic to new architecture. I have 1yr retention period. Reading the documentations I understand that I need to run both for a year to let the old data expire and then get rid of old classic architecture deployment.
Is there anyone in similar situation and find a way to prevent long term dual Mimir deployment?
Are you migrating to the same object storage bucket, or to a new one?
If both the classic cluster and the new ingest-storage cluster point to the same TSDB object storage, then the new cluster’s queriers can read the historical blocks already in that bucket. In that case, you shouldn’t need to keep the classic cluster running for the full 1-year retention period—the official migration guide only requires both clusters to coexist during the migration and cutover process before the old cluster is decommissioned.
If you’re also migrating to a different object storage bucket, then I don’t believe there’s currently a supported way to migrate the historical TSDB blocks into the new ingest-storage deployment. In that scenario, running both environments until the old data expires (or maintaining access to the old cluster for historical queries) may indeed be necessary.
Most likely possible to move data from an object storage to another and reconfigure the cluster to read from new object storage. There will be a downtime though
OK confession time. I have apparently misread the documentation
As stated in the docs and replied by @infofcc3, using same S3 bucket with new Mimir cluster and keeping them for 12 hours approx worked like a charm. I don’t have any gaps with the metrics and now successfully decommissioned old Mimir deployment and fully running on new architecture.
Using the same S3 bucket and keeping both deployments running during the migration worked as expected, and you were able to decommission the old Mimir deployment after the migration without waiting for the full 1-year retention period.
Glad to hear the migration completed successfully with no metric gaps.
