Does an HA Loki instance need persistent local storage?

Grafana Loki
1

I have a Loki cluster. I set up the cluster initially with persistent volumes (via CSI) so that if any instances needed to be restarted (or removed/added) they would have the same persistent storage. This worked great. I have since updated the cluster to use remote object storage. Now I am wondering if I need to keep the persistent volumes for anything.

I was reading through the TSDB documentation and see that Index Caching is not required.

This is how the persistent volume looks on one the instances in my cluster:

loki/
  chunks/           (empty)
  compactor/        (empty)
  index/
    multitenant/
      index_19544/  (created Jul 07)
      ...           (many directories)
      index_19620/  (created today)
    per_tenant/     (empty)
    scratch/        (empty)
    wal/
      1695242727/   (created today)
        00000000    (created today, 7866 bytes)
  rules/            (empty)
  tsdb-cache/
    index_19539     (created today)
    ...             (many directories)
    index_19620     (created today)

Here is my relevant configuration:

        common:
          storage:
            gcs:
              bucket_name: <bucket_name>

        schema_config:
          configs:
            - from: "2022-01-01"
              store: boltdb
              object_store: filesystem
              schema: v11
              index:
                prefix: index_
                period: 168h
            - from: "2023-07-01"
              store: tsdb
              object_store: gcs
              schema: v12
              index:
                prefix: index_
                period: 24h

        storage_config:
          boltdb:
            directory: /loki-volume/loki/index

          filesystem:
            directory: /loki-volume/loki/chunks

          tsdb_shipper:
            active_index_directory: /loki-volume/loki/index
            shared_store: gcs
            cache_location: /loki-volume/loki/tsdb-cache

Would my cluster continue to run properly if I removed:

  • the older schema config
  • the boltdb storage config
  • the filesystem storage config
  • the tsdb_shipper.cache_location
  • the persistent volumes

I assume that the contents of the active_index_directory are ephemeral and will be recreated if the instance is terminated or moved to a different node.

2

  1. You should be able to remove old storage configuration if you are certain that no data exists there anymore.

  2. You do want persistent volumes for the injesters (writer) and compactor (backend if you use simple scalable mode) components.

1 Like
3

@tonyswumac, thank you for the response.

Do you know what would happen to Loki if the ingester and compactor were to only us ephemeral storage?

4

I have been running a cluster of Loki instances for about two hours. They are not under live load, but they seem to be running well. I have storage set to use GCS. The GCS is the only form of persistent storage at the moment. I have removed the persistent volume. Anything that the Loki instances write to local disk will be ephemeral. I have deployed new instances and killed old instances several times and I am not seeing any fatal errors.

5

If you are using write-ahead log (WAL) on the ingester, the logs that’s in WAL will be lost if you don’t use persistent volume. You can try to do flush on shutdown, but I don’t have much experience with that I am not sure how reliable that is.

Compactor writes marker file (the file that records which chunk files to delete according to retention), if you don’t use persistent volume for compactor compactor will miss deleting some chunks. This is not as big of a deal, you can always configure retention on your object storage as well and just delete files after your longest retention configured in Loki. Personally I am not using persistent volume for compactor either.

1 Like