Loki doesn't work

Hi,

I started implementing Loki (static manifest via helm template) when I deployed everything is up and running on my cluster however in every Loki component logs there are errors, I couldn’t identify what potential root causes of these issues, what could be missing in my configuration or misconfigured causing these issues?

LOGS

logs of write :

caller=memberlist_client.go:598 msg="joining memberlist cluster: failed to reach any nodes" retries=4 err="1 error occurred:\n\t* Failed to resolve loki-memberlist: lookup loki-memberlist on 172.**.**.**:**: no such host\n\n""

logs of read :

caller=delete_requests_client.go:211 msg="error getting delete requests from the store" err="unexpected status code: 404 
caller=spanlogger.go:80 user=fake level=error msg="failed loading deletes for user" err="unexpected status code: 404""

logs of gateway :

"172.**.***.*** - - [28/Mar/2023:21:56:34 +0000]  200 "GET / HTTP/1.1" 2 "-" "kube-probe/1.21" "-"" 
"***.**.**.**- self-monitoring [28/Mar/2023:21:56:36 +0000]  101 "GET /loki/api/v1/tail?query=%7Bstream%3D%22stdout%22%2Cpod%3D%22loki-canary-vc8tw%22%7D HTTP/1.1" 38 "-" "Go-http-client/1.1" "-"" 

logs of canary :

"Querying loki for logs with query: http://loki-gateway.monitoring.svc.cluster.local./loki/api/v1/query_range?start=1680038720362276486&end=1680038740362276486&query=%7Bstream%3D%22stdout%22%2Cpod%3D%22loki-canary-vc8tw%22%7D&limit=1000" 
"failed to find entry 1680038730362276486 in Loki when spot check querying 13m47.000485976s after it was written" 

CONTEXT

I specified Loki’s config in Grafana datasources (but nothing shows in the explore list of Grafana)

    - name: Loki
      type: loki
      orgId: 1
      access: proxy
      url: http://loki-gateway.monitoring.svc.cluster.local:80
      version: 2
      jsonData:
        maxLines: 1000
      uid: loki_uid

I created two buckets chunks and ruler and specified their names in this configmap:

  config.yaml: |
    server:
      http_listen_port: 3100
      grpc_listen_port: 9095
      log_level: warn
      http_server_read_timeout: 240s
      http_server_write_timeout: 240s
    auth_enabled: false
    memberlist:
      join_members:
        - loki-memberlist
    analytics:
      reporting_enabled: false
    common:
      path_prefix: /var/loki
      replication_factor: 3
      storage:
        gcs:
          bucket_name: loki_chunks_bucket
          chunk_buffer_size: 0
          request_timeout: "0s"
          enable_http2: true
    storage_config:
      hedging:
        at: 250ms
        max_per_second: 20
        up_to: 3
    schema_config:
      configs:
        - from: 2022-09-01
          store: boltdb-shipper
          object_store: gcs
          schema: v12
          index:
            prefix: loki_index_
            period: 24h
    limits_config:
      enforce_metric_name: false
      reject_old_samples: true
      reject_old_samples_max_age: 168h
      max_cache_freshness_per_query: 10m
      split_queries_by_interval: 15m
    ruler:
      storage:
        type: gcs
        gcs:
          bucket_name: loki_ruler_bucket
          chunk_buffer_size: 0
          request_timeout: "0s"
          enable_http2: true
    ring:
      kvstore:
        store: memberlist
    query_range:
      align_queries_with_step: true
    ingester_client:
      remote_timeout: 15s

ENVIRONMENT

loki : v2.7.0

grafana-operator: v3.0.1

Kubernetes: 1.21.14-gke.14600

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.