How to reduce the number of data chunk in storage

Grafana Loki
1

I deployed Loki through a cluster and used AWS for backend storage. However, I noticed that AWS storage generates between 4000-5000 data blocks per hour (based on the creation time of the data blocks), and at least one data chunk is generated per second.

More than 1.2 million data blocks have been generated in 12 days, and I have tried to adjust many parameters, but the total number cannot be reduced. What should I do to control the number of data blocks

The Loki version I am using is version 3.2.0

The configuration of loki.yml is as follows

auth_enabled: false

memberlist:
  bind_port: 9095
  join_members:
  - *****:9095
  - *****:9095

server:
  http_listen_port: 3100
  http_tls_config:
    cert_file: /*****/loki.crt
    key_file:  /*****/loki.key
    client_ca_file: /*****/ca.crt
    client_auth_type: RequireAndVerifyClientCert
  grpc_listen_port: 3110
  grpc_server_max_recv_msg_size: 109951162777600
  grpc_server_max_send_msg_size: 109951162777600 
  grpc_server_keepalive_timeout: 3600s
  log_level: error 

common:
  ring:
    replication_factor: 1
    kvstore:
      store: memberlist
  path_prefix: /*****/loki-data


ingester:
  chunk_retain_period: 1800s  
  flush_check_period: 3600s 
  max_chunk_age: 3600s 
  chunk_idle_period: 3600s 
  chunk_block_size:  104857600 
  chunk_target_size: 10485760 
  sync_min_utilization: 0.8
  concurrent_flushes: 1
  wal:
    enabled: true
  lifecycler:
    final_sleep: 10s   
    min_ready_duration: 3s

schema_config:
  configs:
  - from: 2020-05-15
    store: tsdb
    object_store: s3
    schema: v13
    index:
      prefix: loki_
      period: 24h
storage_config:
  tsdb_shipper:
    resync_interval: 1m
  aws:
    s3: s3://<*****>:<*****>@<URL>/loki

querier:
  max_concurrent: 30

query_range:
  max_retries: 2

compactor:
  compaction_interval: 60m
  retention_enabled: true
  delete_request_store: aws
  retention_delete_delay: 1m

analytics:
  reporting_enabled: false

limits_config:
  max_global_streams_per_user: 20000
  max_streams_per_user: 100000
  ingestion_rate_strategy: local
  ingestion_rate_mb: 1024
  ingestion_burst_size_mb: 1024
  reject_old_samples: false
  max_query_parallelism: 500
  max_entries_limit_per_query: 0 
  max_concurrent_tail_requests: 200
  increment_duplicate_timestamp: true
  max_line_size: 10240KB
  tsdb_max_query_parallelism: 1280
  split_queries_by_interval:  24h
  query_timeout: 3600m
  split_metadata_queries_by_interval: 10m
  per_stream_rate_limit: 1G
  per_stream_rate_limit_burst: 1G
  max_query_series: 5000
  shard_streams:
    enabled: false
  retention_period: 48h
2
4

here is that to try to use static tags and modify the values of max_chunk_age and chunk_target size, but that’s how I did it.

I also tried configuring 5 Promtail clients with static tags but is still generating chunk every second

  - targets:
    labels:
      app: test1
      ip:  <IP1>
      __path__: /var/log/app3.log
  - targets:
    labels:
      app: test2
      ip: <IP2>
      __path__: /var/log/app3.log
  - targets:
    labels:
      app: test3
      ip: <IP3>
      __path__: /var/log/app3.log```
5

enable debug logging, there can be reason why chunk was flushed (size, timeout,… ). They are your logs, so you should to know them: maybe you really have so much logs, that you are reaching chunk size limit every second. I guess metrics are your friend to observe how Loki components perform and what their load. Maybe you have some weird FS (AWS storage is not usefull definition) - again read debug logs instead of guessings

6

Your max_chunk_age and chunk_idle_period are too small. You can also set bigger target size.

7

Some label have too many values, deleting these label can solve the problem