How to build a loki cluster and query logs instantly

Grafana Loki
#1

The plan is to build the cluster of 3 nodes, one for running queries and two for ingesting data to object storage. All 3 nodes are members of memberlist

memberlist:
  abort_if_cluster_join_fails: false
  bind_port: 7946
  max_join_backoff: 1m
  max_join_retries: 10
  min_join_backoff: 1s
  join_members:
  - loki1.env.infra.domain:7946
  - logproxy1.env.infra.domain:7946
  - logproxy2.env.infra.domain:7946

data is stored in

storage_config:
  boltdb_shipper:
    active_index_directory: "/var/lib/loki/index"
    cache_location: "/var/lib/loki/index_cache"
    resync_interval: 5s
    shared_store: s3
  aws:
    bucketnames: ocsysinfra-loki
    endpoint: datastore.domain
    region: WR
    access_key_id: aKey
    secret_access_key: aSecret
    insecure: false
    sse_encryption: false
    http_config:
      idle_conn_timeout: 90s
      response_header_timeout: 0s
      insecure_skip_verify: true
    s3forcepathstyle: true

All works perfect except whenever I try to run a query on querier loki1.env.infra.domain, the logs come with random delay 2-7 min. Is it the way it just it is? Can I get logs instantly from what was shipped by fluentbit to logproxy1,2.env.infra.domain or is it impossible? I also observing

loki1 loki[4568]: level=error ts=2021-04-13T12:47:58.73859361Z caller=batch.go:667 msg="error fetching chunks" err="context canceled"

whenever I run a query like

logcli-linux-amd64  query '{job="fluentbit"}'  -q | head -1
2021-04-13T13:29:45Z ....

We use latest binary from Release v2.2.1 · grafana/loki · GitHub on Ubuntu.

#2

Unfortunately, I don’t know the answer to your question, but I am intrigued that it seems you are running Loki cluster outside of Kubernetes? Is that true?

If yes, I need some guidance on how to do that as I am unable to find any docs for such.

#3

rizwankh, what exactly you are willing to know? We are not running any production Loki yet, just a test mode. And yes, we use no k8s for it. We use single binary from loki release, logs are sent to loki with fluent bit and then stored to s3. We use grafana to see the logs.

#4

Hello. I have the same problem. I use Loki in HA mode with 2 replicas and s3 as storage. Loki is installed into the k8s cluster. When I use Explore in Grafana to see logs, I get this error in Loki logs:

level=error ts=2021-06-23T13:45:24.805432721Z caller=batch.go:667 msg="error fetching chunks" err="RequestCanceled: request context canceled\ncaused by: context canceled"

How to solve this problem?

#5

Upgraded to loki 2.3, still same problem just the query times out faster

level=error ts=2021-09-01T19:38:38.368688593Z caller=frontend_processor.go:146 msg=“error processing requests” err=EOF
level=error ts=2021-09-01T19:38:38.368466512Z caller=batch.go:690 msg=“error fetching chunks” err=“context canceled”

#6

If you are still interested, get in touch. We are running it on AWS ECS.