How to implementation of Loki Log Backup and Restore on Azure Kubernetes?

Grafana Loki
1

Dear Community,

I am currently facing challenges in implementing a backup and restore solution for Loki logs deployed on Azure Kubernetes, with Loki’s log storage residing in an Azure Storage Account file share. As illustrated in the attached image, I am seeking guidance on an effective approach to address this issue.

image
image2708×888 105 KB

My preliminary plan involves copying the logs stored by Loki on AKS1 (located in fileshare1) to another file share, fileshare2. Subsequently, I would create a new AKS cluster, AKS2, and redeploy Loki and Grafana using the deployment YAML from AKS1. Finally, I would mount fileshare2 as a Persistent Volume Claim (PVC) on AKS2.

Unfortunately, I have encountered issues in practice, as depicted in the attached image, with pod errors indicating an “invalid database.” Despite extensive research, I have been unable to resolve this issue. I kindly request the assistance of the community to help me troubleshoot and address this problem. Your expertise and insights would be highly appreciated.

image
image1906×687 149 KB

Thank you sincerely for your assistance!

2

I don’t have a lot of experiences with Azure Cloud, so apologies if I am way off in my replies.

Firstly, your issue possibly has nothing to do with Loki. I’d recommend you to verify that your replication is actually working.

Secondly, if you are using Kubernetes, which implies that you plan to at least run simple scalable mode, shouldn’t Loki be configured to write to azure storage directly instead of via a mounted file system? Also, wouldn’t it be easier to use azure storage’s replication policy instead of robocopy?