Hi,
I started implementing Loki (static manifest via helm template) when I deployed everything is up and running on my cluster however in every Loki component logs there are errors, I couldn’t identify what potential root causes of these issues, what could be missing in my configuration or misconfigured causing these issues?
LOGS
logs of write :
caller=memberlist_client.go:598 msg="joining memberlist cluster: failed to reach any nodes" retries=4 err="1 error occurred:\n\t* Failed to resolve loki-memberlist: lookup loki-memberlist on 172.**.**.**:**: no such host\n\n""
logs of read :
caller=delete_requests_client.go:211 msg="error getting delete requests from the store" err="unexpected status code: 404
caller=spanlogger.go:80 user=fake level=error msg="failed loading deletes for user" err="unexpected status code: 404""
logs of gateway :
"172.**.***.*** - - [28/Mar/2023:21:56:34 +0000] 200 "GET / HTTP/1.1" 2 "-" "kube-probe/1.21" "-""
"***.**.**.**- self-monitoring [28/Mar/2023:21:56:36 +0000] 101 "GET /loki/api/v1/tail?query=%7Bstream%3D%22stdout%22%2Cpod%3D%22loki-canary-vc8tw%22%7D HTTP/1.1" 38 "-" "Go-http-client/1.1" "-""
logs of canary :
"Querying loki for logs with query: http://loki-gateway.monitoring.svc.cluster.local./loki/api/v1/query_range?start=1680038720362276486&end=1680038740362276486&query=%7Bstream%3D%22stdout%22%2Cpod%3D%22loki-canary-vc8tw%22%7D&limit=1000"
"failed to find entry 1680038730362276486 in Loki when spot check querying 13m47.000485976s after it was written"
CONTEXT
I specified Loki’s config in Grafana datasources (but nothing shows in the explore list of Grafana)
- name: Loki
type: loki
orgId: 1
access: proxy
url: http://loki-gateway.monitoring.svc.cluster.local:80
version: 2
jsonData:
maxLines: 1000
uid: loki_uid
I created two buckets chunks and ruler and specified their names in this configmap:
config.yaml: |
server:
http_listen_port: 3100
grpc_listen_port: 9095
log_level: warn
http_server_read_timeout: 240s
http_server_write_timeout: 240s
auth_enabled: false
memberlist:
join_members:
- loki-memberlist
analytics:
reporting_enabled: false
common:
path_prefix: /var/loki
replication_factor: 3
storage:
gcs:
bucket_name: loki_chunks_bucket
chunk_buffer_size: 0
request_timeout: "0s"
enable_http2: true
storage_config:
hedging:
at: 250ms
max_per_second: 20
up_to: 3
schema_config:
configs:
- from: 2022-09-01
store: boltdb-shipper
object_store: gcs
schema: v12
index:
prefix: loki_index_
period: 24h
limits_config:
enforce_metric_name: false
reject_old_samples: true
reject_old_samples_max_age: 168h
max_cache_freshness_per_query: 10m
split_queries_by_interval: 15m
ruler:
storage:
type: gcs
gcs:
bucket_name: loki_ruler_bucket
chunk_buffer_size: 0
request_timeout: "0s"
enable_http2: true
ring:
kvstore:
store: memberlist
query_range:
align_queries_with_step: true
ingester_client:
remote_timeout: 15s
ENVIRONMENT
loki : v2.7.0
grafana-operator: v3.0.1
Kubernetes: 1.21.14-gke.14600