Query error loki reporting to grafana it can't find chunks

Grafana Loki
1

I’m using the loki-distributed helm chart and trying to use s3 as storage. The s3 storage is working when it’s the single basic loki install but not with distributed.

I’m getting the following error in grafana.

Query error
open /var/loki/chunks/ZmFrZS9lMTdjYTUxMTFmZGE3MzdlOjE3ZjRkNGE5ZjFjOjE3ZjRkNGJhNmVhOjM1ZTg1NTAz: no such file or directory

So the chunk file is there on the ingester but not the querier so it seems like its something with my configs like its not looking in the ingester when it should for recent logs and just hitting the querier?

Below is the config for loki

loki:
  storageConfig:
    aws:
      s3: s3://us-east-1/foobar-loki-storage-test
      region: us-east-1
      s3forcepathstyle: true
    filesystem:
      directory: /var/loki/chunks
    boltdb_shipper:
      active_index_directory: /var/loki/index
      shared_store: aws
      cache_location: /var/loki/cache
      cache_ttl: 2h 
    index_queries_cache_config:
      enable_fifocache: true
      fifocache:
        max_size_bytes: 2GB
        max_size_items: 0
        validity: 24h0m0s
        size: 0

  config: |
    auth_enabled: false
    
    server:
      http_listen_port: 3100

    distributor:
      ring:
        kvstore:
          store: memberlist
    
    memberlist:
      join_members:
        - {{ include "loki.fullname" . }}-memberlist
    
    ingester:
      lifecycler:
        ring:
          kvstore:
            store: memberlist
          replication_factor: 1
      chunk_idle_period: 5m
      chunk_block_size: 262144
      chunk_encoding: snappy
      chunk_retain_period: 1m
      chunk_target_size: 1048576 
      query_store_max_look_back_period: 5m
      wal:
        dir: /var/loki/wal

    limits_config:
      enforce_metric_name: false
      reject_old_samples: true
      reject_old_samples_max_age: 168h
      max_cache_freshness_per_query: 10m
      ingestion_rate_mb: 150
      ingestion_burst_size_mb: 100
      max_streams_per_user: 100000
      max_query_parallelism: 32
      max_entries_limit_per_query: 0

    {{- if .Values.loki.schemaConfig}}
    schema_config:
    {{- toYaml .Values.loki.schemaConfig | nindent 2}}
    {{- end}}

    {{- if .Values.loki.storageConfig}}
    storage_config:
    {{- toYaml .Values.loki.storageConfig | nindent 2}}
    {{- if .Values.indexGateway.enabled}}
      index_gateway_client:
        server_address: dns:///{{ include "loki.indexGatewayFullname" . }}:9095
    {{- end}}
    {{- end}}

    chunk_store_config:
      max_look_back_period: 0s

    table_manager:
      retention_deletes_enabled: false
      retention_period: 0s

    querier:
      engine:
        timeout: 5m
      query_ingesters_within: 1h
      query_timeout: 5m

    query_range:
      align_queries_with_step: true
      max_retries: 5
      split_queries_by_interval: 15m
      cache_results: true
      results_cache:
        cache:
          enable_fifocache: true
          fifocache:
            max_size_items: 1024
            validity: 24h
    
    frontend_worker:
      frontend_address: {{ include "loki.queryFrontendFullname" . }}:9095
    
    frontend:
      log_queries_longer_than: 5s
      compress_responses: true
      tail_proxy_url: http://{{ include "loki.querierFullname" . }}:3100
    
    compactor:
      shared_store: aws
      compaction_interval: 5m
    
    ruler:
      storage:
        type: local
        local:
          directory: /etc/loki/rules
      ring:
        kvstore:
          store: memberlist
2

So this is a working helm chart. The big issue was thinking i needed filesystem still and indexGateway isn’t enabled by default so it had no clue where to route requests for data.

Also it seems when that all is enabled the querier breaks for volume issues

loki:
  containerSecurityContext:
    readOnlyRootFilesystem: false
  
  config: |
    auth_enabled: false
    server:
      http_listen_port: 3100
    distributor:
      ring:
        kvstore:
          store: memberlist
    memberlist:
      join_members:
        - {{ include "loki.fullname" . }}-memberlist
    ingester:
      lifecycler:
        ring:
          kvstore:
            store: memberlist
          replication_factor: 1
      chunk_idle_period: 30m
      chunk_block_size: 262144
      chunk_encoding: snappy
      chunk_retain_period: 1m
      max_transfer_retries: 0
      wal:
        dir: /var/loki/wal
    limits_config:
      enforce_metric_name: false
      reject_old_samples: true
      reject_old_samples_max_age: 168h
      max_cache_freshness_per_query: 10m
  
    schema_config:
      configs:
      - from: 2020-09-07
        store: boltdb-shipper
        object_store: filesystem
        schema: v11
        index:
          prefix: loki_index_
          period: 24h

    storage_config:
      aws:
        s3: s3://us-east-1/company-loki-storage-test
        region: us-east-1
        s3forcepathstyle: true
      boltdb_shipper:
        shared_store: s3
        active_index_directory: /var/loki/index
        cache_location: /var/loki/cache
        cache_ttl: 168h
        index_gateway_client:
          server_address: dns:///{{ include "loki.indexGatewayFullname" . }}:9095

    chunk_store_config:
      max_look_back_period: 0s
    table_manager:
      retention_deletes_enabled: false
      retention_period: 0s
    query_range:
      align_queries_with_step: true
      max_retries: 5
      split_queries_by_interval: 15m
      cache_results: true
      results_cache:
        cache:
          enable_fifocache: true
          fifocache:
            max_size_items: 1024
            validity: 24h
    frontend_worker:
      frontend_address: {{ include "loki.queryFrontendFullname" . }}:9095
    frontend:
      log_queries_longer_than: 5s
      compress_responses: true
      tail_proxy_url: http://{{ include "loki.querierFullname" . }}:3100
    compactor:
      working_directory: /var/loki/compactor
      shared_store: s3
    ruler:
      storage:
        type: local
        local:
          directory: /etc/loki/rules
      ring:
        kvstore:
          store: memberlist
      rule_path: /tmp/loki/scratch

serviceAccount:
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::11111111111:role/loki-test

gateway:
  serverSnippet: "client_max_body_size 100M;"

indexGateway:
  enabled: true

  persistence:
    enabled: true

querier:
  replicas: 1
  extraVolumes:
    - name: bolt-db
      emptyDir: {}
  extraVolumeMounts:
    - name: bolt-db
      mountPath: /var/loki
3

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.