Time until loki streams are searchable by querier

Grafana Loki
,
1

In my environment (Grafana version 11.1.0 / Loki version 3.1.0) low volume streams take longer to be searchable by querier compared to high volume streams. This is especially problematic for very low volume streams (1 event per day or less) where it can take more than 24 hours until the event is found by the querier. It seems like the ingester is not properly queried by the querier. As I send the same event to two different log-aggregation system, I know that the event was sent without delay. query_ingesters_within is set to 0 (but also tried 96h). Any help is appreciated.

querier:
  query_ingesters_within: 0 
  max_concurrent: 24
  concurrent_flushes: 32
  flush_check_period: 30s
  flush_op_backoff:
    min_period: 10s
    max_period: 1m0s
    max_retries: 10
  flush_op_timeout: 10m0s
  chunk_retain_period: 0s
  chunk_idle_period: 30m0s
  chunk_block_size: 262144
  chunk_target_size: 1572864
  chunk_encoding: snappy
  max_chunk_age: 2h0m0s
  autoforget_unhealthy: false
  sync_period: 1h0m0s
  sync_min_utilization: 0.1
  max_returned_stream_errors: 10
  query_store_max_look_back_period: 2h41m0s

  wal:
    enabled: true

query_scheduler:
  max_outstanding_requests_per_tenant: 32000

limits_config:
  max_global_streams_per_user: 0
  shard_streams:
    enabled: false
  discover_service_name:
  discover_log_levels: false
  reject_old_samples: false
  query_timeout: 30s

schema_config:
  configs:
    - from: 2024-05-01
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h
2
  1. How are you deploying Loki?
  2. Do you have a server section of your Loki configuration, or is that it?
3

Hi @tonyswumac
Loki is deployed in single binary mode.
Below is the complete loki config.yml

auth_enabled: false

server:
  http_listen_port: 3101
  grpc_listen_port: 9097
  grpc_server_max_recv_msg_size: 20971520
  grpc_server_max_send_msg_size: 20971520 
internal_server:
  http_listen_port: 3111

ingester:
  wal:
    enabled: true
    dir: /opt/loki/loki_default/wal
  chunk_encoding: snappy

common:
  instance_addr: 127.0.0.1
  path_prefix: /opt/loki/loki_default
  storage:
    filesystem:
      chunks_directory: /opt/loki/loki_default/chunks
      rules_directory: /opt/loki/loki_default/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

querier:
  query_ingesters_within: 0
  max_concurrent: 24

query_scheduler:
  max_outstanding_requests_per_tenant: 32000

limits_config:
  ingestion_burst_size_mb: 4096
  ingestion_rate_mb: 2048
  per_stream_rate_limit: 1024M
  per_stream_rate_limit_burst: 2048M
  retention_period: 90d
  max_global_streams_per_user: 0
  shard_streams:
    enabled: false
  discover_service_name:
  discover_log_levels: false
  reject_old_samples: false
  query_timeout: 30s

schema_config:
  configs:
    - from: 2024-05-01
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h

compactor:
  working_directory: /opt/loki/loki_default/compactor
  retention_enabled: true
  delete_request_store: filesystem

table_manager:
  retention_deletes_enabled: true
  retention_period: 90d
4

Try this:

  1. Change instance_addr to 0.0.0.0.
  2. Add a memberlist configuration, with your Loki instance’s IP. This may not be necessary, I’d try with just #1 first and see if that fixes it.