Simple Scalable on VM's not working

Hi,
I have setup Loki as a monolithic service on a VM and that worked fine.
Now I am trying to set up Loki simple scalable solution and have set up a load balancer which direct write requests to a write Loki instance (target=write) and all other requests to the read instance (target=read).
When I try to add the Loki datasource to Grafana I always get the following error in the Grafana UI Unable to fetch labels from Loki (Failed to call resource), please check the server logs for more details.
From Loki logs:

Nov 28 10:26:10 loki loki[27458]: level=warn ts=2022-11-28T10:26:10.771657041Z caller=logging.go:86 traceID=361df086f6551498 orgID=fake msg="GET /loki/api/v1/labels?start=1669630570765000000&end=1669631170765000000 (500) 7.566374ms Response: \"rpc error: code = Unimplemented desc = unknown service logproto.Querier\\n\" ws: false; Accept-Encoding: gzip; User-Agent: Grafana/9.2.4; "

and in the Grafana logs:
Nov 28 10:26:10 grafana grafana-server[75502]: logger=context userId=1 orgId=1 uname=admin t=2022-11-28T10:26:10.771909503Z level=error msg="Failed to call resource" error="rpc error: code = Unimplemented desc = unknown service logproto.Querier\n" traceID=

My Loki config is as follows (the target is set as read/write respectively)

auth_enabled: false
target: read #run Loki in single binary mode

server:
  http_listen_port: 3100
  grpc_listen_port: 9096

frontend:
  max_outstanding_per_tenant: 2048 #max number of requests http 429 if exceeded

common:
  path_prefix: /etc/loki
  replication_factor: 1
  ring:
    instance_addr: 127.0.0.1
    kvstore:
      store: memberlist
  storage:
    s3:
      region: eu-west-2
      bucketnames: loki-cluster

memberlist:
  bind_port: 7946
  join_members:
    - loki-memberlist:7946 

schema_config:
  configs:
    - from: 2022-11-06
      store: boltdb-shipper
      object_store: s3
      schema: v11
      index:
        prefix: loki-dev_
        period: 24h

storage_config:
  aws:
    s3: s3://eu-west-2/loki-cluster
  boltdb_shipper:
    active_index_directory: /etc/loki/boltdb-shipper-active
    cache_location: /etc/loki/boltdb-shipper-cache
    cache_ttl: 24h
    shared_store: s3

In addition to this, when I set the storage to use as S3, the write node does not immediately send the data to S3, it starts writing to the wal directory and takes a long time before anything appears in S3 (apart from the loki_cluster_seed.json file). It only synced to S3 after 1 hour. Any ideas why this is?

It looks like the issue is with the querying as I get the following errors when running queries via Grafana K6:

Nov 28 11:13:37 loki loki[27531]: level=info ts=2022-11-28T11:13:37.173527769Z caller=table_manager.go:262 msg="query readiness setup completed" duration=2.548µs distinct_users_len=0
Nov 28 11:16:07 loki loki[27531]: ts=2022-11-28T11:16:07.804266486Z caller=spanlogger.go:80 user=fake level=info org_id=fake latency=fast query_type=labels length=1h0m0s duration=692.034µs status=500 label= throughput=0B total_bytes=0B total_entries=0
Nov 28 11:16:07 loki loki[27531]: level=error ts=2022-11-28T11:16:07.804587438Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="rpc error: code = Code(500) desc = rpc error: code = Unimplemented desc = unknown service logproto.Querier\n"
Nov 28 11:16:07 loki loki[27531]: ts=2022-11-28T11:16:07.805598207Z caller=spanlogger.go:80 user=fake level=info org_id=fake latency=fast query_type=labels length=1h0m0s duration=547.093µs status=500 label= throughput=0B total_bytes=0B total_entries=0
Nov 28 11:16:07 loki loki[27531]: level=error ts=2022-11-28T11:16:07.805916469Z caller=retry.go:73 org_id=fake msg="error processing request" try=1 err="rpc error: code = Code(500) desc = rpc error: code = Unimplemented desc = unknown service logproto.Querier\n"
Nov 28 11:16:07 loki loki[27531]: ts=2022-11-28T11:16:07.806904919Z caller=spanlogger.go:80 user=fake level=info org_id=fake latency=fast query_type=labels length=1h0m0s duration=526.059µs status=500 label= throughput=0B total_bytes=0B total_entries=0
Nov 28 11:16:07 loki loki[27531]: level=error ts=2022-11-28T11:16:07.807184374Z caller=retry.go:73 org_id=fake msg="error processing request" try=2 err="rpc error: code = Code(500) desc = rpc error: code = Unimplemented desc = unknown service logproto.Querier\n"
Nov 28 11:16:07 loki loki[27531]: ts=2022-11-28T11:16:07.808092829Z caller=spanlogger.go:80 user=fake level=info org_id=fake latency=fast query_type=labels length=1h0m0s duration=489.119µs status=500 label= throughput=0B total_bytes=0B total_entries=0

Turns out it is an issue with the firewall not allowing port 9096. All good now.

1 Like