we are running one Loki instance version v.2.6.1 (monolith deployment type, filesystem(EBS)) in monitoring namespace. Recently we have deployed to the same name namespace Loki v3.0.0 in Simple Scalable Mode with s3 backend. When all properly set up we are planning to delete the older Loki v2. Is it possible that they somehow interfering to each other? Because we can see a lot of s3 related errors like:
failed to load chunk ‘fake/3a126a38213da3a3/1900c3a00f0:1900ca80fef:160f2692’: failed to get s3 object: NoSuchKey: The specified key does not exist. status code: 404, request id: xxx, host id: xxx
Any advice or guidance would be greatly appreciated.
How does the compatibility between Loki version v2.6.1 (monolith deployment with EBS filesystem) and Loki v3.0.0 (Simple Scalable Mode with S3 backend) impact their simultaneous operation?
If your two clusters aren’t using the same storage (sounds like v2 is filesystem and v3 is S3) then I don’t think they would interferer with each other. Have you checked and see if the object in the error log exists or not?
Also, did you deploy the two clusters in the same namespace? I’d double check and make sure they aren’t connecting to each other and forming a membership ring somehow.
Hi, yes, that’s exactly what happened. The older deployment instance joined the ring. When stopped the older instance looks like all issues are gone. Thank you for the info.
