View Log Latency

Grafana Loki
1

The latest log can be viewed within 5 minutes, the log cannot be viewed within 5-30 minutes, and the log can also be viewed before 30 minutes.

2

The picture is like this

image
image700×187 2.89 KB

3

A bit more information on your setup and configuration would be helpful.

4 
auth_enabled: false
chunk_store_config:
  chunk_cache_config:
    background:
      writeback_buffer: 100000
      writeback_goroutines: 100
    embedded_cache:
      enabled: false
    memcached:
      batch_size: 2048
      expiration: 72h
      parallelism: 1000
    memcached_client:
      addresses: dnssrv+_memcached-client._tcp.prod-loki-loki-distributed-memcached-chunks.log.svc.cluster.local
      consistent_hash: true
      timeout: 500ms
  max_look_back_period: 0s
  write_dedupe_cache_config:
    background:
      writeback_buffer: 100000
      writeback_goroutines: 100
    memcached:
      batch_size: 2048
      expiration: 72h
      parallelism: 1000
    memcached_client:
      addresses: dnssrv+_memcached-client._tcp.prod-loki-loki-distributed-memcached-index-writes.log.svc.cluster.local
      consistent_hash: true
      timeout: 500ms
common:
  compactor_address: http://prod-loki-loki-distributed-compactor:3100
compactor:
  compaction_interval: 10m
  retention_delete_delay: 2h
  retention_delete_worker_count: 150
  retention_enabled: true
  shared_store: s3
  shared_store_key_prefix: index/
  working_directory: /data/loki/boltdb-shipper-compactor
distributor:
  ring:
    kvstore:
      store: memberlist
frontend:
  compress_responses: true
  log_queries_longer_than: 30s
  max_outstanding_per_tenant: 2048
  scheduler_address: prod-loki-loki-distributed-query-scheduler:9095
  tail_proxy_url: http://prod-loki-loki-distributed-querier:3100
frontend_worker:
  parallelism: 100
  scheduler_address: prod-loki-loki-distributed-query-scheduler:9095
ingester:
  chunk_block_size: 524288
  chunk_encoding: snappy
  chunk_idle_period: 30m
  chunk_retain_period: 10m
  chunk_target_size: 4194304
  concurrent_flushes: 64
  flush_check_period: 15s
  flush_op_timeout: 10m
  lifecycler:
    ring:
      kvstore:
        store: memberlist
      replication_factor: 1
  max_transfer_retries: 0
  wal:
    dir: /var/loki/wal
ingester_client:
  grpc_client_config:
    max_recv_msg_size: 2097152000
    max_send_msg_size: 2097152000
  remote_timeout: 1s
limits_config:
  cardinality_limit: 200000
  enforce_metric_name: false
  ingestion_burst_size_mb: 65536
  ingestion_rate_mb: 32768
  max_cache_freshness_per_query: 10m
  max_entries_limit_per_query: 1000000
  max_global_streams_per_user: 100000
  max_label_name_length: 10240
  max_label_names_per_series: 300
  max_label_value_length: 20480
  max_query_lookback: 90d
  max_query_parallelism: 500
  per_stream_rate_limit: 2048MB
  per_stream_rate_limit_burst: 4096MB
  reject_old_samples: false
  reject_old_samples_max_age: 168h
  retention_period: 91d
  split_queries_by_interval: 15m
memberlist:
  join_members:
  - prod-loki-loki-distributed-memberlist
querier:
  max_concurrent: 50
  query_ingesters_within: 2h
  query_timeout: 60s
query_range:
  align_queries_with_step: true
  cache_results: true
  max_retries: 5
  results_cache:
    cache:
      background:
        writeback_buffer: 100000
        writeback_goroutines: 100
      memcached:
        batch_size: 2048
        expiration: 72h
        parallelism: 1000
      memcached_client:
        consistent_hash: true
        host: prod-loki-loki-distributed-memcached-frontend
        service: memcached-client
        timeout: 500ms
query_scheduler:
  grpc_client_config:
    max_recv_msg_size: 2097152000
    max_send_msg_size: 2097152000
  max_outstanding_requests_per_tenant: 2048
ruler:
  alertmanager_url: https://alertmanager.xx
  external_url: https://alertmanager.xx
  ring:
    kvstore:
      store: memberlist
  rule_path: /tmp/loki/scratch
  storage:
    s3:
      access_key_id: ----------
      bucketnames: loki
      endpoint: ----------
      http_config:
        idle_conn_timeout: 90s
        insecure_skip_verify: true
        response_header_timeout: 0s
      insecure: true
      region: us
      s3forcepathstyle: false
      secret_access_key: -------------
      sse_encryption: false
    type: s3
runtime_config:
  file: /var/loki-distributed-runtime/runtime.yaml
schema_config:
  configs:
  - from: "2020-09-07"
    index:
      period: 24h
      prefix: loki_index_
    object_store: s3
    schema: v11
    store: boltdb-shipper
server:
  grpc_server_max_concurrent_streams: 0
  grpc_server_max_recv_msg_size: 2097152000
  grpc_server_max_send_msg_size: 2097152000
  http_listen_port: 3100
  log_level: info
storage_config:
  aws:
    access_key_id: ----------
    bucketnames: loki
    endpoint: --------
    http_config:
      idle_conn_timeout: 90s
      insecure_skip_verify: true
      response_header_timeout: 0s
    insecure: true
    region: ------
    s3forcepathstyle: false
    secret_access_key: ----------
    sse_encryption: false
  boltdb_shipper:
    active_index_directory: /var/loki/index
    cache_location: /var/loki/cache
    cache_ttl: 168h
    index_gateway_client:
      server_address: dns:///prod-loki-loki-distributed-index-gateway:9095
    shared_store: s3
  index_queries_cache_config:
    background:
      writeback_buffer: 100000
      writeback_goroutines: 100
    memcached:
      batch_size: 2048
      expiration: 72h
      parallelism: 1000
    memcached_client:
      addresses: dnssrv+_memcached-client._tcp.prod-loki-loki-distributed-memcached-index-queries.log.svc.cluster.local
      consistent_hash: true
      timeout: 500ms
table_manager:
  retention_deletes_enabled: true
  retention_period: 100d
5

Use helm deployed and memcache cluster

6

When the log volume is small, there is no problem. When the log volume of a single application reaches a certain level, it will reproduce stably.

7

  1. Since you set query_ingesters_within to 2h, I’d recommend setting ingester.max_chunk_age to 2h as well.

  2. How many ingesters are you running? Check the /ring endpoint of your ingesters and make sure all members are present.

  3. This is probably not related, but table manager is no longer needed if you enable compactor, so I’d recommend turning it off with:

table_manager:
  retention_deletes_enabled: false
8
  1. Found ingester.max_chunk_age the default is 2h
  2. ingesters is all members are present.
  3. I don’t quite understand why the table history deletion management is closed. Isn’t compression only improving performance?
9

Will it have anything to do with chunk_retain_period: 10m

10

  1. Can you post a screenshot of your /ring screenshot?

  2. Any interesting log from querier? Enabling debug log may be beneficial as well.

  3. As mentioned, table manager is not necessary if compactor is running. See Retention | Grafana Loki documentation (again this is unrelated to your issue most likely, you don’t have to fix if you don’t want to).

11

Thanks，My problem has been solved，replication_factor set 1 ，Now I set 3，It is suspected that setting 1 may not be handled and will have an impact during query.

12

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.