Filesystem retention causes error

Grafana Loki
1

Hi,

I’m using filesystem as a store and want to set a retention period of 31 days. It seems that after setting the below configuration, the older logs are not accessible anymore, however I am seeing this error in the log files. When I remove the Limits Config and Compacter, the error is disappears again, but that would disable the retention.

What am I missing in this configuration?

I am using Loki 3.0

level=error ts=2024-09-19T16:38:25.0660799Z caller=gennumber_loader.go:136 msg="error loading cache generation numbers" err="rpc error: code = Unavailable desc = connection error: desc = \"error reading server preface: http2: frame too large\""
level=error ts=2024-09-19T16:38:25.0660799Z caller=gennumber_loader.go:136 msg="error loading cache generation numbers" err="rpc error: code = Unavailable desc = connection error: desc = \"error reading server preface: http2: frame too large\""
level=error ts=2024-09-19T16:38:25.0660799Z caller=gennumber_loader.go:136 msg="error loading cache generation numbers" err="rpc error: code = Unavailable desc = connection error: desc = \"error reading server preface: http2: frame too large\""
level=error ts=2024-09-19T16:38:25.0660799Z caller=gennumber_loader.go:136 msg="error loading cache generation numbers" err="rpc error: code = Unavailable desc = connection error: desc = \"error reading server preface: http2: frame too large\""
level=error ts=2024-09-19T16:38:25.0671536Z caller=gennumber_loader.go:136 msg="error loading cache generation numbers" err="rpc error: code = Unavailable desc = connection error: desc = \"error reading server preface: http2: frame too large\""

ts=2024-09-19T16:43:27.7574077Z caller=spanlogger.go:109 user=fake level=error msg="failed loading deletes for user" err="rpc error: code = Unavailable desc = connection error: desc = \"error reading server preface: http2: frame too large\""
ts=2024-09-19T16:43:27.7574077Z caller=spanlogger.go:109 user=fake level=error msg="failed loading deletes for user" err="rpc error: code = Unavailable desc = connection error: desc = \"error reading server preface: http2: frame too large\""

Configuration:

auth_enabled: false

server:
#  http_listen_address: 0.0.0.0
  http_listen_port: 3100
  grpc_listen_port: 0

common:
  instance_addr: 127.0.0.1
  path_prefix: D:\Data\Loki
  storage:
    filesystem:
      chunks_directory: D:\Data\Loki\chunks
      rules_directory: D:\Data\Loki\rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory
      
storage_config:
  filesystem:
    directory: D:\Data\Loki\chunks

query_range:
  results_cache:
    cache:
      embedded_cache:
        enabled: true
        max_size_mb: 1000

schema_config:
  configs:
    - from: 2020-10-24
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h
        
# retention configuration
compactor:
  working_directory: D:\Data\Loki
  compaction_interval: 10m
  retention_enabled: true
  retention_delete_delay: 2h
  retention_delete_worker_count: 150
  delete_request_store: filesystem

limits_config:
  retention_period: 744h
  allow_deletes: true
  deletion_mode: "filter-and-delete"

ruler:
  alertmanager_url: http://localhost:9093

frontend:
  encoding: protobuf
2

Maybe try tweaking the message size configuration:

server:
  # 100MB
  grpc_server_max_recv_msg_size: 1.048576e+08
  grpc_server_max_send_msg_size: 1.048576e+08
3

Thank you for the suggestion, unfortunately it did not resolve the issue. I’ve increased the size to 500MB and still getting the same error. Also removed the disable port of GRPC (which was set to 0). GZIP compression is also enabled for GRPC

server:
#  http_listen_address: 0.0.0.0
  http_listen_port: 3100
  grpc_server_max_recv_msg_size: 524288000
  grpc_server_max_send_msg_size: 524288000

I’m not sure what else there could be…

4

I am not sure either. I’ll test your configuration next week when I can, but I don’t see anything obvious. I should also point out that I don’t use Windows, so it’s possible that it’s a problem only for Windows. If you have any Linux server might be a good thing to try in the meantime.