Level=error msg="failed to flush user"

[janisi]

Hello, we are using S3 bucket for storing logs with Loki (and promtail) on our K8s cluster. We see a lot of error messages in Loki logs msg="failed to flush user".

Did we hit some S3 limitations or it’s something with configuration? We don’t have any errors on our development cluster with the same exact Loki configuration.

Thanks, any help will be appreciated.

level=error ts=2023-03-01T06:56:01.687692933Z caller=flush.go:221 org_id=fake msg="failed to flush user" err="RequestCanceled: request context canceled\ncaused by: context deadline exceeded"
level=error ts=2023-03-01T06:56:01.687737296Z caller=flush.go:221 org_id=fake msg="failed to flush user" err="RequestCanceled: request context canceled\ncaused by: context deadline exceeded"
level=error ts=2023-03-01T06:56:01.860268489Z caller=flush.go:221 org_id=fake msg="failed to flush user" err="RequestCanceled: request context canceled\ncaused by: context deadline exceeded"

Loki config: loki.yaml (grafana/loki:2.4.2)

auth_enabled: false
chunk_store_config:
  max_look_back_period: 744h
compactor:
  retention_enabled: true
  shared_store: s3
  working_directory: /data/loki/boltdb-shipper-compactor
ingester:
  chunk_block_size: 262144
  chunk_idle_period: 5m
  chunk_retain_period: 30s
  lifecycler:
    ring:
      kvstore:
        store: inmemory
      replication_factor: 1
  max_chunk_age: 5m
  max_transfer_retries: 0
  wal:
    dir: /data/loki/wal
limits_config:
  enforce_metric_name: false
  reject_old_samples: true
  reject_old_samples_max_age: 168h
  retention_period: 744h
schema_config:
  configs:
  - from: "2021-06-14"
    index:
      period: 24h
      prefix: loki_index_
    object_store: aws
    schema: v11
    store: boltdb-shipper
server:
  http_listen_port: 3132
storage_config:
  aws:
    region: nl-ams
    s3: https://ACCESS:TOKEN@s3.nl-ams.scw.cloud/bucket_name
    s3forcepathstyle: true
  boltdb_shipper:
    active_index_directory: /data/loki/index
    cache_location: /data/loki/boltdb-cache
    cache_ttl: 24h
    shared_store: s3
  filesystem:
    directory: /data/loki/chunks
table_manager:
  retention_deletes_enabled: true
  retention_period: 744h

[tonyswumac]

Double check your configuration, i believe when you refer to shared storage it should be shared_store: aws instead of shared_store: s3.

Also, what’s your log volume like? What’s your average chunk size? We are currently writing about 80 chunks per second with roughly a quarter of them at full size (which should be roughly 150MB), and we haven’t had any issue.