Loki: On premise deployment

Grafana Loki
1

Hi there,

I’m working on the Loki chart: loki/production/helm/loki at main · grafana/loki · GitHub

My setup:

  • On premise cluster with 6 nodes
  • Remote Ceph cluster, I have a storageClass that’s taking care of the provisioning, all I need to do is Claim (Either Block storage or FS).

Issue:
It seems that it’s mandatory to pick a cloud storage provider: loki/production/helm/loki/values.yaml at 0065fd6e95fc7531abf3d3d8aab33ec0f8aeea8f · grafana/loki · GitHub

Even if I null the s3 settings, the backend is trying to connect to a bucket and spams errors that it can’t find the credentials.

Questions:

  1. Is there a way to simply use my Ceph storage instead of a cloud storage?
  2. I’m a bit confused as to how the HA loki is working, there’s Read, Write and Backend, understandably Backend is meant to storage the logs, index them etc, and Write to simply do the writes.

My understanding is that I need Block storage solution for the Backend, and simply buffer the writes until the logs are written in the backend?

My storage config:

backend:
persistence:
volumeClaimsEnabled: true
dataVolumeParameters:
emptyDir: {}
enableStatefulSetAutoDeletePVC: false
size: 64Gi
storageClass: csi-rbd-sc
selector: null

2

Hello!

We also run Loki on premise and we have Ceph in use. For Ceph there’s a gateway daemon you can run that supports S3 compatible object storage with Ceph. We use this gateway for our Loki setup as well. An object storage for Loki is needed if you want to run it in HA mode (which you probably want).

For further details see:
https://docs.ceph.com/en/quincy/radosgw/index.html

However depending on your configuration regarding compression, you might want to tune this gateway accordingly and give it enough bandwidth to transmit the data.

1 Like
3

Thank you @ndominko that’s a very solid answer, really appreciate that.

1- I didn’t know Ceph has Object storage solution, I was looking into MinIO
2- I was hoping that I could do NFS/CephFS

Essentially my initial understanding, that I can PVC in the pod and store, index and read the logs from there is wrong.

4

None of the Loki components store permanent data. Both indices and chunks are written to object storage as permanent storage. The persistent volume is needed for only write and backend due to special needs (write needs it to store WAL logs, backend needs it to store delete marker files).