Grafana is stoping quering Loki

[holosian]

Im getting in to such problem. After installation of Promtail and Loki (with helm loki-distributed) on different servers everything is working. But after few hours Grafana is stoping querying Loki with error:

Query error
object not found in storage

On Ingester logs i can find such logs:

level=info ts=2021-11-05T15:38:58.407920033Z caller=table_manager.go:171 msg="uploading tables"
level=info ts=2021-11-05T15:39:58.408565657Z caller=table_manager.go:171 msg="uploading tables"
level=info ts=2021-11-05T15:40:58.408057869Z caller=table_manager.go:171 msg="uploading tables"
level=info ts=2021-11-05T15:41:58.408098741Z caller=table_manager.go:171 msg="uploading tables"
level=info ts=2021-11-05T15:42:58.408413197Z caller=table_manager.go:171 msg="uploading tables"
level=info ts=2021-11-05T15:43:58.408576382Z caller=table_manager.go:171 msg="uploading tables"
level=info ts=2021-11-05T15:44:58.40890704Z caller=table_manager.go:171 msg="uploading tables"
level=info ts=2021-11-05T15:45:58.408766835Z caller=table_manager.go:171 msg="uploading tables"
level=info ts=2021-11-05T15:46:58.407908413Z caller=table_manager.go:171 msg="uploading tables"
level=info ts=2021-11-05T15:47:58.40858152Z caller=table_manager.go:171 msg="uploading tables"
level=info ts=2021-11-05T15:47:58.43226484Z caller=table.go:336 msg="uploading table loki_index_18936"
level=info ts=2021-11-05T15:47:58.43229784Z caller=table.go:363 msg="finished uploading table loki_index_18936"
level=info ts=2021-11-05T15:47:58.43230788Z caller=table.go:421 msg="cleaning up unwanted dbs from table loki_index_18936"
level=info ts=2021-11-05T15:48:58.407916491Z caller=table_manager.go:171 msg="uploading tables"
level=info ts=2021-11-05T15:48:58.408088121Z caller=table.go:336 msg="uploading table loki_index_18936"
level=info ts=2021-11-05T15:48:58.408106051Z caller=table.go:363 msg="finished uploading table loki_index_18936"
level=info ts=2021-11-05T15:48:58.408158621Z caller=table.go:421 msg="cleaning up unwanted dbs from table loki_index_18936"
level=info ts=2021-11-05T15:49:58.40874014Z caller=table_manager.go:171 msg="uploading tables"
level=info ts=2021-11-05T15:49:58.44394924Z caller=table.go:336 msg="uploading table loki_index_18936"
level=info ts=2021-11-05T15:49:58.44400212Z caller=table.go:363 msg="finished uploading table loki_index_18936"

when that “cleaning” log appear Grafana is stoping querying.

After ingester restart everything is working again for some time.What configuration im missing i set up PVC for ingester and problem still appear.

[holosian]

I needed to configure boltdb and GCS bucket to store chunks and indexex and its strted to work:

    configs:
    - from: 2020-09-07
      store: boltdb-shipper
      object_store: gcs
      schema: v11
      index:
        prefix: loki_index_
        period: 24h

  # -- Check https://grafana.com/docs/loki/latest/configuration/#storage_config for more info on how to configure storages
  storageConfig:
    boltdb_shipper:
      shared_store: gcs
      active_index_directory: /var/loki/index
      cache_location: /var/loki/cache
      cache_ttl: 168h
    filesystem:
      directory: /var/loki/chunks
    gcs:
      bucket_name: grafana-loki-bucket