I’ve installed loki and promtail using the helm charts but grafana can’t read and promtail can’t write.
I’m using k3s:
kubectl version --short
Client Version: v1.25.4
Kustomize Version: v4.5.7
Server Version: v1.24.9+k3s2
10.42.4.126 - self-monitoring [19/Feb/2023:13:12:18 +0000] 499 "GET /loki/api/v1/query_range?start=1676805899181497587&end=1676812277182072987&query=%7Bstream%3D%22stdout%22%2Cpod%3D%22loki-canary-zllpv%22%7D&limit=1000 HTTP/1.1" 0 "-" "loki-canary/2.7.3" "-"
10.42.4.128 - - [19/Feb/2023:13:12:23 +0000] 499 "POST /loki/api/v1/push HTTP/1.1" 0 "-" "GrafanaAgent/" "-"
10.42.5.144 - - [19/Feb/2023:13:12:26 +0000] 499 "POST /loki/api/v1/push HTTP/1.1" 0 "-" "GrafanaAgent/" "-"
10.42.5.1 - - [19/Feb/2023:13:12:27 +0000] 200 "GET / HTTP/1.1" 2 "-" "kube-probe/1.24" "-"
10.42.4.128 - - [19/Feb/2023:13:12:34 +0000] 499 "POST /loki/api/v1/push HTTP/1.1" 0 "-" "GrafanaAgent/" "-"
10.42.5.1 - - [19/Feb/2023:13:12:37 +0000] 200 "GET / HTTP/1.1" 2 "-" "kube-probe/1.24" "-"
10.42.3.199 - - [19/Feb/2023:13:12:41 +0000] 499 "POST /loki/api/v1/push HTTP/1.1" 0 "-" "GrafanaAgent/" "-"
10.42.4.128 - - [19/Feb/2023:13:12:44 +0000] 499 "POST /loki/api/v1/push HTTP/1.1" 0 "-" "GrafanaAgent/" "-"
10.42.5.1 - - [19/Feb/2023:13:12:47 +0000] 200 "GET / HTTP/1.1" 2 "-" "kube-probe/1.24" "-"
10.42.3.198 - self-monitoring [19/Feb/2023:13:12:51 +0000] 499 "GET /loki/api/v1/tail?query=%7Bstream%3D%22stdout%22%2Cpod%3D%22loki-canary-wlc8x%22%7D HTTP/1.1" 0 "-" "Go-http-client/1.1" "-"
10.42.4.128 - - [19/Feb/2023:13:12:56 +0000] 499 "POST /loki/api/v1/push HTTP/1.1" 0 "-" "GrafanaAgent/" "-"
10.42.5.1 - - [19/Feb/2023:13:12:57 +0000] 200 "GET / HTTP/1.1" 2 "-" "kube-probe/1.24" "-"
10.42.5.144 - - [19/Feb/2023:13:13:00 +0000] 499 "POST /loki/api/v1/push HTTP/1.1" 0 "-" "GrafanaAgent/" "-"
Here is what the helm charts set up:
kubectl get all -n monitoring
NAME READY STATUS RESTARTS AGE
pod/grafana-loki-grafana-agent-operator-6945fc54d7-xdn6k 1/1 Running 0 114m
pod/grafana-loki-logs-9g56b 2/2 Running 0 114m
pod/grafana-loki-logs-csw5n 2/2 Running 0 114m
pod/grafana-loki-logs-mdbhm 2/2 Running 0 114m
pod/grafana-promtail-4b87g 1/1 Running 0 18m
pod/grafana-promtail-7fknd 1/1 Running 0 19m
pod/grafana-promtail-cgzpr 1/1 Running 0 25m
pod/grafana-promtail-f8dpq 1/1 Running 0 19m
pod/grafana-promtail-nkvhr 1/1 Running 0 19m
pod/grafana-promtail-np9tr 1/1 Running 0 19m
pod/loki-canary-thwlv 1/1 Running 0 114m
pod/loki-canary-wlc8x 1/1 Running 0 114m
pod/loki-canary-zllpv 1/1 Running 0 114m
pod/loki-gateway-5d468df8-268bp 1/1 Running 0 42m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/loki-canary ClusterIP 10.43.249.205 <none> 3500/TCP 114m
service/loki-gateway ClusterIP 10.43.247.226 <none> 80/TCP 43m
service/loki-memberlist ClusterIP None <none> 7946/TCP 114m
service/loki-read ClusterIP 10.43.104.220 <none> 3100/TCP,9095/TCP 114m
service/loki-read-headless ClusterIP None <none> 3100/TCP,9095/TCP 114m
service/loki-write ClusterIP 10.43.161.201 <none> 3100/TCP,9095/TCP 114m
service/loki-write-headless ClusterIP None <none> 3100/TCP,9095/TCP 114m
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
daemonset.apps/grafana-loki-logs 3 3 3 3 3 <none> 114m
daemonset.apps/grafana-promtail 6 6 6 6 6 <none> 77m
daemonset.apps/loki-canary 3 3 3 3 3 <none> 114m
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/grafana-loki-grafana-agent-operator 1/1 1 1 114m
deployment.apps/loki-gateway 1/1 1 1 114m
NAME DESIRED CURRENT READY AGE
replicaset.apps/grafana-loki-grafana-agent-operator-6945fc54d7 1 1 1 114m
replicaset.apps/loki-gateway-5b5994874b 0 0 0 46m
replicaset.apps/loki-gateway-5d468df8 1 1 1 114m
NAME READY AGE
statefulset.apps/loki-read 0/3 114m
statefulset.apps/loki-write 0/3 114m
The only values I changed were my storage class:
loki:
persistence:
enabled: true
storageClass: cephfs
and the tolerations in promtail:
tolerations:
- key: CriticalAddonsOnly
operator: Exists
So it is a pretty much a vanilla install.
What am I missing?