Is it possible to increase loki long queries speed?

Hello.

query with range 30 days takes about 5 minutes. VM uses only 2-3 Gb of RAM, but 8Gb is available. Is it possible to increase query speed ?
(grafana and loki(with BoltDB) are deployed in docker containers on the same VM)

#logcli query  "{project_env=\"PROD\"} |= \"Salesforce query new Opportunity\"" --stats    --since=720h

Ingester.TotalReached 		 32
Ingester.TotalChunksMatched 	 236
Ingester.TotalBatches 		 0
Ingester.TotalLinesSent 	 0
Ingester.HeadChunkBytes 	 847 kB
Ingester.HeadChunkLines 	 4344
Ingester.DecompressedBytes 	 1.4 MB
Ingester.DecompressedLines 	 4943
Ingester.CompressedBytes 	 190 kB
Ingester.TotalDuplicates 	 0
Store.TotalChunksRef 		 128496
Store.TotalChunksDownloaded 	 128496
Store.ChunksDownloadTime 	 4m5.609681266s
Store.HeadChunkBytes 		 0 B
Store.HeadChunkLines 		 0
Store.DecompressedBytes 	 5.9 GB
Store.DecompressedLines 	 23955333
Store.CompressedBytes 		 512 MB
Store.TotalDuplicates 		 0
Summary.BytesProcessedPerSecond 	 41 MB
Summary.LinesProcessedPerSecond 	 167820
Summary.TotalBytesProcessed 		 5.9 GB
Summary.TotalLinesProcessed 		 23964620
Summary.ExecTime 			 2m22.798728993s

loki config:

auth_enabled: false

server:
  http_listen_port: 3100
  http_server_read_timeout: 600s
  http_server_write_timeout: 600s
  log_level: debug
  grpc_server_max_recv_msg_size: 20388078
  grpc_server_max_send_msg_size: 20388078

ingester:
  lifecycler:
    address: 127.0.0.1
    ring:
      kvstore:
        store: inmemory
      replication_factor: 1
    final_sleep: 0s
  wal:
    enabled: true
    dir: /loki/wal
  chunk_idle_period: 1h       # Any chunk not receiving new logs in this time will be flushed
  max_chunk_age: 1h           # All chunks will be flushed when they hit this age, default is 1h
  chunk_target_size: 3145728  # Loki will attempt to build chunks up to 1.5MB, flushing first if chunk_idle_period or max_chunk_age is reached first
  chunk_retain_period: 20m    # Must be greater than index read cache TTL if using an index cache (Default index read cache TTL is 5m)
  max_transfer_retries: 0     # Chunk transfers disabled

schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h
      chunks:
        prefix: chunk_
        period: 24h

storage_config:
  boltdb_shipper:
    active_index_directory: /loki/boltdb-shipper-active
    cache_location: /loki/boltdb-shipper-cache
    cache_ttl: 720h         # Can be increased for faster performance over longer query periods, uses more disk space
    shared_store: filesystem
  filesystem:
    directory: /loki/chunks

compactor:
  working_directory: /loki/boltdb-shipper-compactor
  shared_store: filesystem
  retention_enabled: true

limits_config:
  enforce_metric_name: false
  reject_old_samples: true
  reject_old_samples_max_age: 168h
  max_query_parallelism: 64
  max_streams_per_user: 500000
  ingestion_rate_mb: 16
  ingestion_burst_size_mb: 24
  retention_period: 31d
  per_stream_rate_limit: 16MB
  per_stream_rate_limit_burst: 64MB
  max_cache_freshness_per_query: '20m'

ruler:
  storage:
    type: local
    local:
      directory: /loki/rules
  rule_path: /loki/rules-temp
  alertmanager_url: http://localhost:9093
  ring:
    kvstore:
      store: inmemory
  enable_api: true

querier:
  max_concurrent: 2048
  query_timeout: 5m
  engine:
    timeout: 5m
  query_ingesters_within: 1h

query_range:
  align_queries_with_step: true
  split_queries_by_interval: 1h
  max_retries: 2
  cache_results: true
  parallelise_shardable_queries: true

  results_cache:
    cache:
      enable_fifocache: true
      fifocache:
        max_size_items: 2048000000
        validity: 720h
        size: 1024

frontend:
    address: 127.0.0.1
    log_queries_longer_than: 5s
    compress_responses: true
    max_outstanding_per_tenant: 2048

query_scheduler:
  max_outstanding_requests_per_tenant: 4096

increasing of parameter query_range: split_queries_by_interval: 24h decreases total time to 1-1.3 min (for 30 days range)
(previously 4.5-5 min)

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.