Hi,
I have setup Loki as a monolithic service on a VM and that worked fine.
Now I am trying to set up Loki simple scalable solution and have set up a load balancer which direct write requests to a write Loki instance (target=write) and all other requests to the read instance (target=read).
When I try to add the Loki datasource to Grafana I always get the following error in the Grafana UI Unable to fetch labels from Loki (Failed to call resource), please check the server logs for more details.
From Loki logs:
Nov 28 10:26:10 loki loki[27458]: level=warn ts=2022-11-28T10:26:10.771657041Z caller=logging.go:86 traceID=361df086f6551498 orgID=fake msg="GET /loki/api/v1/labels?start=1669630570765000000&end=1669631170765000000 (500) 7.566374ms Response: \"rpc error: code = Unimplemented desc = unknown service logproto.Querier\\n\" ws: false; Accept-Encoding: gzip; User-Agent: Grafana/9.2.4; "
and in the Grafana logs:
Nov 28 10:26:10 grafana grafana-server[75502]: logger=context userId=1 orgId=1 uname=admin t=2022-11-28T10:26:10.771909503Z level=error msg="Failed to call resource" error="rpc error: code = Unimplemented desc = unknown service logproto.Querier\n" traceID=
My Loki config is as follows (the target is set as read/write respectively)
auth_enabled: false
target: read #run Loki in single binary mode
server:
http_listen_port: 3100
grpc_listen_port: 9096
frontend:
max_outstanding_per_tenant: 2048 #max number of requests http 429 if exceeded
common:
path_prefix: /etc/loki
replication_factor: 1
ring:
instance_addr: 127.0.0.1
kvstore:
store: memberlist
storage:
s3:
region: eu-west-2
bucketnames: loki-cluster
memberlist:
bind_port: 7946
join_members:
- loki-memberlist:7946
schema_config:
configs:
- from: 2022-11-06
store: boltdb-shipper
object_store: s3
schema: v11
index:
prefix: loki-dev_
period: 24h
storage_config:
aws:
s3: s3://eu-west-2/loki-cluster
boltdb_shipper:
active_index_directory: /etc/loki/boltdb-shipper-active
cache_location: /etc/loki/boltdb-shipper-cache
cache_ttl: 24h
shared_store: s3
In addition to this, when I set the storage to use as S3, the write node does not immediately send the data to S3, it starts writing to the wal directory and takes a long time before anything appears in S3 (apart from the loki_cluster_seed.json file). It only synced to S3 after 1 hour. Any ideas why this is?
It looks like the issue is with the querying as I get the following errors when running queries via Grafana K6:
Nov 28 11:13:37 loki loki[27531]: level=info ts=2022-11-28T11:13:37.173527769Z caller=table_manager.go:262 msg="query readiness setup completed" duration=2.548µs distinct_users_len=0
Nov 28 11:16:07 loki loki[27531]: ts=2022-11-28T11:16:07.804266486Z caller=spanlogger.go:80 user=fake level=info org_id=fake latency=fast query_type=labels length=1h0m0s duration=692.034µs status=500 label= throughput=0B total_bytes=0B total_entries=0
Nov 28 11:16:07 loki loki[27531]: level=error ts=2022-11-28T11:16:07.804587438Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="rpc error: code = Code(500) desc = rpc error: code = Unimplemented desc = unknown service logproto.Querier\n"
Nov 28 11:16:07 loki loki[27531]: ts=2022-11-28T11:16:07.805598207Z caller=spanlogger.go:80 user=fake level=info org_id=fake latency=fast query_type=labels length=1h0m0s duration=547.093µs status=500 label= throughput=0B total_bytes=0B total_entries=0
Nov 28 11:16:07 loki loki[27531]: level=error ts=2022-11-28T11:16:07.805916469Z caller=retry.go:73 org_id=fake msg="error processing request" try=1 err="rpc error: code = Code(500) desc = rpc error: code = Unimplemented desc = unknown service logproto.Querier\n"
Nov 28 11:16:07 loki loki[27531]: ts=2022-11-28T11:16:07.806904919Z caller=spanlogger.go:80 user=fake level=info org_id=fake latency=fast query_type=labels length=1h0m0s duration=526.059µs status=500 label= throughput=0B total_bytes=0B total_entries=0
Nov 28 11:16:07 loki loki[27531]: level=error ts=2022-11-28T11:16:07.807184374Z caller=retry.go:73 org_id=fake msg="error processing request" try=2 err="rpc error: code = Code(500) desc = rpc error: code = Unimplemented desc = unknown service logproto.Querier\n"
Nov 28 11:16:07 loki loki[27531]: ts=2022-11-28T11:16:07.808092829Z caller=spanlogger.go:80 user=fake level=info org_id=fake latency=fast query_type=labels length=1h0m0s duration=489.119µs status=500 label= throughput=0B total_bytes=0B total_entries=0