Getting throttled by s3

Grafana Loki
1

I am getting errors from S3 bucket for throttling . failed to flush user" err="SlowDown: Please reduce your request rate.\n\tstatus code: 503. Is this handled by Loki ?

2

Hi,

There are some tuning you can do check yhr ingestion_burst_size and ingestion_rate_mb, also note that when you just implement there is an overhead since loki indexing a lot of data.

Hope it helps.

Good Luck

3

S3 throttles by prefix within a bucket, so if you have the ability to use Loki in multi-tenant mode, this will put each of the chunks in a tenant prefix.

Other considerations are to make sure you aren’t sending really high cardinality data, more streams means more chunks which means more IOPS to S3.

You could paste your config also and I could check if you have anything misconfigured.

1 Like
4

config: |
auth_enabled: false

server:
  log_level: debug
  # Must be set to 3100
  http_listen_port: 3100
  grpc_server_max_recv_msg_size: 67108864
  grpc_server_max_send_msg_size: 67108864

distributor:
  ring:
    kvstore:
      store: memberlist

ingester:
  # Disable chunk transfer which is not possible with statefulsets
  # and unnecessary for boltdb-shipper
  max_transfer_retries: 0
  chunk_idle_period: 1h
  chunk_target_size: 1536000
  max_chunk_age: 1h
  flush_check_period: 60s
  lifecycler:
    join_after: 0s
    ring:
      replication_factor: 2
      kvstore:
        store: memberlist

memberlist:
  join_members:
    - {{ include "loki.fullname" . }}-memberlist

limits_config:
  ingestion_rate_mb: 10
  ingestion_burst_size_mb: 20
  max_concurrent_tail_requests: 20
  max_cache_freshness_per_query: 10m

schema_config:
  configs:
    - from: 2020-09-07
      store: boltdb-shipper
      object_store: aws
      schema: v11
      index:
        prefix: loki_index_
        period: 24h

chunk_store_config:
  max_look_back_period: 336h

table_manager:
  retention_deletes_enabled: true
  retention_period: 336h

storage_config:
  aws:
    s3: s3://ap-southeast-1/swiggy-loki
  boltdb_shipper:
    active_index_directory: /var/loki/index
    shared_store: s3
    cache_location: /var/loki/cache

query_range:
  # make queries more cache-able by aligning them with their step intervals
  align_queries_with_step: true
  max_retries: 5
  # parallelize queries in 15min intervals
  split_queries_by_interval: 15m
  cache_results: true

  results_cache:
    cache:
      enable_fifocache: true
      fifocache:
        max_size_items: 1024
        validity: 24h

frontend_worker:
  frontend_address: {{ include "loki.queryFrontendFullname" . }}:9095

frontend:
  log_queries_longer_than: 5s
  compress_responses: true

compactor:
  shared_store: s3
5

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.