Queries don't work in a loki-distributed deployment

Hello,

I have deployed the loki-distributed helm chart, with boltdb-shipper and filesystem storage, with only one instance of each component, as docs recommend for filesystem storage.

Ingestion i think is ok, i can see chunks under /var/loki/chunks in ingester container. Only 2 labels, host and app. Apps with a large amount of events, queries get the error that “it cant find chunk, file doesnt exist”, but it is not true, the file exists in /var/loki/chunks in the ingester. The apps with fewer events i can query last 5 or 15 minutes, but if I increased the period of time i get the same error.

Is this normal? Maybe the distributed deployment doesnt work with filesystem storage … or not? It is strange, because this is the default configuration of the chart.

Something i have seen, is that the querier doesnt use the persistent storage, under /var/loki only an empty chunks folder.

Tomorrow i will explore the possibility of deploy minio on the kubernetes cluster with local persistent volume.

Regards,
Ivan

When running with file system storage there are a couple of things you’ll want to tweak. You can also search my post history, I’ve had discussions with others running on file system before.

  1. You’ll want to make sure your system is configured with enough file handles limit.
  2. You’ll want to configure your querier to NOT split queries, or split them at a long interval such as 12h or 24h.
  3. You’ll want to tweak your chunk limits so that you end up writing bigger and fewer chunks (to avoid strain on your filesystem, especially when querying). You can do this by setting bigger chunk size, longer chunk idle time, and longer max chunk age.

Thanks Tony,

I have change:

    ingester:
      chunk_block_size: 2621440
      chunk_target_size: 15728640
      max_chunk_age: 168h
      chunk_idle_period: 168h
      chunk_retain_period: 168h
    limits_config:
      split_queries_by_interval: 24h

And it seems i can query more data, but i keep on not being able to search in all hosts and apps, the same error with the chunks. I will wait until tomorow to see the result of the changes.

Regards,
Ivan