Loki space issue

[mmallikarjuna]

Loki keep complaining about space issue, how could be fix this filling up space. Error: rpc error: code = Unknown desc = mkdir /loki/boltdb-cache/loki_index_19327: no space left on device

[b0b]

Hi @mmallikarjuna ,

This answer is a bit late but maybe it will help someone. This was possibly caused by having /loki on a read-only filesystem. If this is for some sort of container runtime platform like e.g. Kubernetes, then you could mount an empty volume at /loki and that should allow for the folder to be created.

Example from my setup

Here you can see the working-directory set

      containers:
        - name: compactor
          image: docker.io/grafana/loki:2.7.3
          imagePullPolicy: IfNotPresent
          args:
            - -config.file=/etc/loki/config/config.yaml
            - -target=compactor
            - -boltdb.shipper.compactor.working-directory=/var/loki/compactor
            - -config.expand-env=true

I add this to make the folder usable

...
          volumeMounts:
            - name: data
              mountPath: /var/loki
...
      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: data-loki-distributed-compactor
...

Just realised that uses a pvc but it would be very similar with non-persistent storage

...
          volumeMounts:
            - name: data
              mountPath: /var/loki
...
        - name: data
          emptyDir: {}
...

Hope that helps