Loki migration between k8s clusters

Grafana Loki
,
1

Hi Loki team.
I’m using loki-distributed on AWS EKS, trying to test for migration between k8s clusters.
I thought that it could be done like what I attached.
But, even though throw /flush API to ingesters in old Loki, a query to new Loki doesn’t get logs that exist in old Loki.
Did I miss something?
Is there the best way to migrate between k8s clusters?
Thanks.

image
image1289×532 76.3 KB

image
image1046×259 41.3 KB

2

image
image1151×125 20.7 KB

image
image1155×95 28.3 KB

3

Could anyone please help me out?
I haven’t solved this issue yet.

4

@nutslove we are facing the same issue, do you have any resolution for the same?

5

@riddhibatas
Have you joined Loki community slack?
You can see what I did from url below.

6

@nutslove thank you, it worked

1 Like
7

I could migrate Loki as procedure below.

  1. Detach legacy Loki gateway from NLB Target.
  2. Delete legacy Loki gateway.
    ※detaching the gateway from NLB was not enough to kill the session from promtail to legacy Loki.
  3. curl -X POST to all ingesters /flush endpoint.
  4. (After a few minutes after 3.) Delete all legacy Loki stack.
  5. Deploy new Loki to the new EKS cluster.
  6. Attach new Loki to NLB.
    The procedure above should be done in 8.5 minutes or promtail drop after all retries.
    ※Or raise backoff_config.max_retries in promtail.

And I created bash below to do above.

#!/bin/bash
# Usage: sh loki_ingester_flush.sh <Name of Realese>
# e.g. sh loki_ingester_flush.sh multi-tenant
if [[ $1 =~ [a-z]+ ]]; then
  echo "Current Context ----> `kubectl config current-context`"
  echo "Helm Realese Name ----> $1"
  read -p "Input 'yes' or 'y' key to start: " process
else
  echo -e "helm realese name is required as argument\n e.g. 'sh loki_ingester_flush.sh multi-tenant'" ## -eオプションはエスケープ文字を解釈するという意味で、改行コードを解釈させるため
  exit 1
fi

if [ $process == "yes" -o $process == "y" ]; then
  INGESTER_POD_IP=`kubectl get po -l app.kubernetes.io/component=ingester,app.kubernetes.io/instance=${1} -n monitoring -o json | jq -r '.items[].status.podIP'`

  kubectl delete deploy ${1}-loki-distributed-gateway -n monitoring

  sleep 10

  for ip in $INGESTER_POD_IP
  do
    echo "curl -X POST http://${ip}:3100/flush"
    curl -X POST http://${ip}:3100/flush
  done

  echo "sleep for 2min to flush chunk in the ingesters..."
  sleep 120

  echo "uninstall ${1} Loki stack"
  helm uninstall $1 -n monitoring
else
  echo "Goodbye!"
  exit 1
fi