Loki using Garage S3 storage 3 nodes cluster

Grafana Loki
1

I am trying to configure a 3 nodes cluster (not using Kubernetes) with Loki using Garage as the S3 storage:

  • Each node has a Garage process. Garage clustering is configured with replication_factor=3 and consistent_mode=consistent. Each node has been assigned a different zone in the Garage layout, so it behaves as a multimaster cluster (each node is capable of reading/writing and data is kept in sync in all the nodes).
  • Each node has a Loki monolithic deployment. Loki clustering has been configured with replication_factor=3 and a ring/memberlist. In each node, Loki has been configured to connect to its local Garage. A load balancer distributes the incoming requests to each Loki instance in a round-robin fashion.

The question is: Can this setup corrupt the storage?

How does the Loki cluster avoid writng the same chunk to two different Garage instances at the same time?

Should I configure Garage in active-passive mode with a virtual floating IP instead, to avoid corruption?

2

I don’t think it’ll corrupt the storage. The filename of the chunk files should be unique, I believe when forming the filename and directory path it uses fingerprint, start time and end time of the chunk, as well as checksum. I don’t remember how the fingerprint is pulled, but as long as your instances have different hostnames I’d imagine it would be ok. You can check the logs for the file name written and confirm.

The only concern when clustering monolithic Loki would be compactor. You’ll want to make sure compactor is clustered as well, so that you don’t have multiple compacting operations happening at the same time.