Having issue with starting the loki write pods

Grafana Loki
1

Hello,

I have installed the latest Loki via helm chart and all is up except the write pods which are throwing constantly these errors:

level=error ts=2024-05-03T23:06:15.377732431Z caller=flush.go:152 component=ingester org_id=fake msg="failed to flush" err="failed to flush chunks: store put chunk: mkdir fake: read-only file system, num_chunks: 1, labels: {app=\"kube-proxy\", container=\"kube-proxy\", filename=\"/var/log/pods/kube-system_kube-proxy-r9tzx_0c67e2f3-b6c6-4510-9ec0-082b771a4801/kube-proxy/0.log\", job=\"kube-system/kube-proxy\", namespace=\"kube-system\", node_name=\"mailu-slave-09be\", pod=\"kube-proxy-r9tzx\", service_name=\"kube-proxy\", stream=\"stderr\"}"
level=info ts=2024-05-03T23:06:15.37776388Z caller=flush.go:176 component=ingester msg="flushing stream" user=fake fp=f5014dcc64f90d6d immediate=true num_chunks=1 labels="{app=\"loki\", component=\"write\", container=\"loki\", filename=\"/var/log/pods/monitoring_loki-write-1_8029db8b-9ad4-4d75-98bc-fb863e1751e4/loki/0.log\", instance=\"logger\", job=\"monitoring/loki\", namespace=\"monitoring\", node_name=\"mailu-slave-09be\", pod=\"loki-write-1\", service_name=\"loki\", stream=\"stderr\"}"
level=error ts=2024-05-03T23:06:15.377785008Z caller=flush.go:152 component=ingester org_id=fake msg="failed to flush" err="failed to flush chunks: store put chunk: mkdir fake: read-only file system, num_chunks: 1, labels: {app=\"loki\", component=\"memcached-chunks-cache\", container=\"memcached\", filename=\"/var/log/pods/monitoring_logger-loki-chunks-cache-0_26b24e6a-c042-4120-a7c3-a260d3f3feec/memcached/0.log\", instance=\"logger\", job=\"monitoring/loki\", namespace=\"monitoring\", node_name=\"mailu-slave-207a\", pod=\"logger-loki-chunks-cache-0\", service_name=\"loki\", stream=\"stderr\"}"
level=error ts=2024-05-03T23:06:15.377792907Z caller=flush.go:152 component=ingester org_id=fake msg="failed to flush" err="failed to flush chunks: store put chunk: mkdir fake: read-only file system, num_chunks: 59, labels: {app=\"mailu-master-main\", component=\"etcd\", container=\"etcd\", filename=\"/var/log/pods/kube-system_etcd-mailu-master-main_64be207f3d25632ca0fd8515708669f4/etcd/0.log\", job=\"kube-system/mailu-master-main\", namespace=\"kube-system\", node_name=\"mailu-master-main\", pod=\"etcd-mailu-master-main\", service_name=\"mailu-master-main\", stream=\"stderr\"}"
level=info ts=2024-05-03T23:06:15.377834183Z caller=flush.go:176 component=ingester msg="flushing stream" user=fake fp=fa875e9282384cab immediate=true num_chunks=1 labels="{app=\"loki\", component=\"memcached-chunks-cache\", container=\"memcached\", filename=\"/var/log/pods/monitoring_logger-loki-chunks-cache-0_26b24e6a-c042-4120-a7c3-a260d3f3feec/memcached/0.log\", instance=\"logger\", job=\"monitoring/loki\", namespace=\"monitoring\", node_name=\"mailu-slave-207a\", pod=\"logger-loki-chunks-cache-0\", service_name=\"loki\", stream=\"stderr\"}"

What can I do to fix it?

2

Says your writer volume is read-only. I’d start there and try to fix that.

3

Not sure which volume is that.

I have quite a few:

Mounts:
  /etc/loki/config from config (rw)
  /etc/loki/runtime-config from runtime-config (rw)
  /var/loki from data (rw)
  /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-4svhj (ro)