The logs of Loki are not shown or partly shown in Grafana.
My environment is Docker Desktop with Kubernetes running on it. I used the Loki stack to install it. I can connect via either http://loki:3100 or via http://loki.grafana-loki.svc.cluster.local.
- In Grafana, the datasource config of Loki gives frequently an error like
Failed to load log volume for this query
parse error at line 1, col 67: syntax error: unexpected IDENTIFIER
- In the Promtail logs I can see:
2024-10-03 19:36:11 level=warn ts=2024-10-03T17:36:11.959668723Z caller=logging.go:86 traceID=549a9dd21956c96a orgID=fake msg="POST /loki/api/v1/push (500) 254.011µs Response: "at least 1 live replicas required, could only find 0 - unhealthy instances: 10.1.2.214:9095\n" ws: false; Content-Length: 410; Content-Type: application/x-protobuf; User-Agent: promtail/2.9.3; "
2024-10-03 19:36:12 level=error ts=2024-10-03T17:36:12.478371512Z caller=compactor.go:374 msg=“error asking ring for who should run the compactor, will check again” err=“at least 1 healthy replica required, could only find 0 - unhealthy instances: 10.1.2.214:9095”
In the Promtail dashboard I can see that logs are scraped.
The Kubernetes I see the following pods run:
NAME READY STATUS RESTARTS AGE
loki-0 1/1 Running 0 97s
loki-grafana-7c95854c5c-sx2g2 2/2 Running 0 97s
loki-promtail-5pfnc 1/1 Running 0 97s
I can Grafana datasource Loki I see via the label “namespace” only the “grafana-loki”, “kube-system” and “argocd” namespaces, not the default.
When selecting the default namespace, which is not in the list, then I get only the error of the unexpected IDENTIFIER and no logs at all.
I frequently get the message in Grafana “Failed to load log volume for this query
parse error at line 1, col 73: syntax error: unexpected IDENTIFIER”
I can see some logs.