Loki, Mimir and Grafana migration

Grafana Loki
, ,
1

  • What Grafana version and what operating system are you using?

    • name: mimir-distributed, version: “v3.1.0”
    • name: grafana, version: “v6.43.4”
    • name: “loki-distributed”, version: 0.58.0
    • Running on k8s GKE 1.23.0
    • Postgres 12.0

  • What are you trying to achieve?
    Hi, we want to migrate our infrastructure to a new k8s cluster and bucket. Loki and Mimir will have a new dedicated bucket in a different project and Grafana will have a new Postgres DB. Thank you.

  • How are you trying to achieve it?

    • Grafana: we dumped the old DB and restored it in a new DB instance, deployed (via Helm) Grafana in the new k8s cluster and attached it to the new DB instance
    • Loki: we copied the old bucket content in a new bucket, deployed (via Helm) Loki in the new k8s cluster and configured it to read from the new bucket
    • Mimir: same steps as Loki

  • What happened?

    • Grafana: the migration seemed to be correct
    • Loki: here, we had some problems: we have several k8s clusters writing to the same bucket (same tenant) but some timeseries were overwritten by the copies, thus generating errors in Grafana visualization with the error “Failed to load log volume for this query”.
    • Mimir: here, we had some problems: we have several k8s clusters writing to the same bucket (same tenant) but some timeseries were overwritten by the copies, thus generating errors in Grafana visualization with the error “Validation failed the consistency check failed because some blocks were not queried (err-mimir-store-consistency-check-failed). The non-queried blocks are: 01GNNG744NV8CA8JH7EJWYT65T 01GNTM3R1S1TZKSBFXZ60YATJ6 01GNDQNFYAMJPVR8P8DKW3YTHE 01GNGA2MVS384Y5XYSB3SD3Y30”.

  • What did you expect to happen?
    Migrated data is visualized correctly in the new Loki/Mimir/Grafana stack.

  • Can you copy/paste the configuration(s) that you are having problems with?
    Default Helm values, only bucket/DB credentials are different.

  • Did you receive any errors in the Grafana UI or in related logs? If so, please tell us exactly what they were.
    See in “What happened?”.

  • Did you follow any online instructions? If so, what is the URL?
    Not found any instructions online on how to migrate from an old cluster (and bucket and DB) to a new one.

My question is: how can I “drain” the old stack, migrate it to a new cluster/bucket/DB without compromising the data consistency?

2

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.