Failed to enable query frontend using simple scalable deployment mode

[andrewk2]

I am currently using simple scalable deployment mode ( 3 reads, 6 write, Helm Chart version: 3.7.0 ) for my logging system. As I have to ingest around 2TB data per day in Loki, the search performance is not sufficient (Summary.BytesProcessedPerSecond 409 MB) for production.

I’ve read blog posts “How to improve Loki Performance” and was tried to follow the step in the doc (Query Frontend | Grafana Loki documentation), I can successfully run a query-frontend pod, but failed in connecting to the Loki read pods.

Besides, according to the Architecture from the doc (Deployment modes | Grafana Loki documentation), query frontend seems to be part of the read pods.

So, my questions are, how to enable the query frontend in my case? Or only distributed deployment mode achieves the goal?

Here’s my config.yaml

auth_enabled: false
common:
  path_prefix: /var/loki
  replication_factor: 1
  storage:
    s3:
      access_key_id: access_key_id
      bucketnames: bucketnames
      endpoint: s3.region.amazonaws.com
      http_config:
        idle_conn_timeout: 600s
      insecure: false
      region: region
      s3forcepathstyle: false
      secret_access_key: secret_access_key
ingester:
  autoforget_unhealthy: true
  chunk_encoding: snappy
  chunk_idle_period: 30m
  chunk_retain_period: 0s
  chunk_target_size: 1572864
  flush_check_period: 15s
  max_transfer_retries: 0
limits_config:
  enforce_metric_name: false
  ingestion_burst_size_mb: 100
  ingestion_rate_mb: 25
  max_cache_freshness_per_query: 10m
  max_query_parallelism: 16
  per_stream_rate_limit: 15mb
  per_stream_rate_limit_burst: 30mb
  reject_old_samples: true
  reject_old_samples_max_age: 720h
  retention_period: 720h
  split_queries_by_interval: 15m
memberlist:
  join_members:
  - loki-memberlist
querier:
  engine:
    max_look_back_period: 30m
    timeout: 30m
query_range:
  align_queries_with_step: true
query_scheduler:
  scheduler_ring:
    instance_interface_names:
    - lo
ruler:
  storage:
    s3:
      access_key_id:  access_key_id
      bucketnames: bucketnames
      endpoint: s3.region.amazonaws.com
      insecure: false
      region: region
      s3forcepathstyle: false
      Secret_access_key: secret_access_key
    type: s3
schema_config:
  configs:
  - from: "2022-12-11"
    index:
      period: 24h
      prefix: loki_index_
    object_store: s3
    schema: v12
    store: boltdb-shipper
server:
  grpc_listen_port: 9098
  grpc_server_max_recv_msg_size: 83386080
  grpc_server_max_send_msg_size: 83386080
  http_listen_port: 3100
  http_server_idle_timeout: 1800s
  http_server_read_timeout: 1800s
  http_server_write_timeout: 1800s
storage_config:
  hedging:
    at: 250ms
    max_per_second: 20
    up_to: 3

# config for query frontend
  frontend:
    max_outstanding_per_tenant: 10240
    log_queries_longer_than: 15s
    compress_responses: true
    downstream_url: querier.log-server-prod.svc.cluster.local:3100
  frontend_worker:
    frontend_address: querier.log-server-prod.svc.cluster.local:3100
    grpc_client_config:
        max_send_msg_size: 10485760
    parallelism: 1
  query_range:
    split_queries_by_interval: 15m
    align_queries_with_step: true
    max_retries: 5

Thanks!

[tonyswumac]

Query frontend operates in one of two modes, push (configured with frontend.downstream_url ) or pull (configured with frontend_worker.frontend_address ). You have both, so that’s a problem. The documentation seems to recommend pull mode, and that’s what we are using. You can read more here: Query Frontend | Grafana Loki documentation

Also, you need to run a set of containers / pods specifically for query frontend with -target=query-frontend as well.