Error connecting loki data source to kube-prometheus-stack

Grafana
1

Hi Team :wave:

I have installed Grafana using kube-prometheus-stack Helm chart under monitor namespace exposed over ingress.

Installed loki using monolithic Helm chart and see the pods are up and running.

Facing issue when adding loki as data source under Grafana with connection URL as http://loki.loki:3100, seeing error message as Unable to connect with Loki. Please check the server logs for more details.

Please help me debug the issue.

2

Hi,

What namespace do you have Loki installed in? Can you confirm that you have loki service in loki namespace (kubectl get svc -n loki). If not, that’s why your connection string is not working. Local service address is built in <service name>.<service namespace> way.

3

Hi @dawiddebowski

Thanks for the reply.

I have installed loki in the namespace loki and the service is up and running

$ kubectl get svc -n loki
NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)              AGE
loki               ClusterIP   10.132.12.240    <none>        3100/TCP,9095/TCP    44h
4

When you go into grafana’s pod kubectl exec -it <grafana pod name> -- /bin/bash can you execute ping loki.loki? If not, can you do the same with loki.loki.svc.cluster.local?

5

Hi @dawiddebowski

I got error as permission denied when tried to ping.

Tried using curl command to loki service which failed

$curl -v loki.loki
* Host loki:80 was resolved.
* IPv6: (none)
* IPv4: 10.110.4.184
*   Trying 10.110.4.184:80...

Tried using curl command to http://loki-gateway.loki which successeded from grafana pod

curl -v loki-monitoring.loki
* Host loki-gateway.loki:80 was resolved.
* IPv6: (none)
* IPv4: 10.110.151.38
*   Trying 10.110.151.38:80...
* Connected to loki-gateway.loki(10.110.151.38) port 80
> GET / HTTP/1.1
> Host: loki-gateway.loki
> User-Agent: curl/8.9.1
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: nginx/1.27.2
< Date: Sat, 05 Oct 2024 12:50:07 GMT
< Content-Type: application/octet-stream
< Content-Length: 2
< Connection: keep-alive
<
* Connection #0 to host loki-gateway.loki left intact
OK
6

Ok, have you tried to check the second (gateway) address in Grafana datasource? Plus curl on the first one shouldn’t also have the port in url?

7

I used the URL as http://loki-gateway.loki:3100 seeing the same error as Unable to connect with Loki. Please check the server logs for more details. :frowning_face:

8

one more thing, I do not find any gateway resource created in the loki namespace when deployed.

I have nginx-gateway running in the cluster

kubectl get gatewayclass
NAME    CONTROLLER                                   ACCEPTED   AGE
nginx   gateway.nginx.org/nginx-gateway-controller   True       85m
9

Ok, I think I’ve got it. Got the same error as you did in UI. Found the issue on Github. Although it was another error (I got no org id in grafana logs, disabling authentication worked. Probably preferred solution would be to figure out how the authentication works but if you’re not exposing loki externally, I don’t think it matters that much :smile:

10

thanks for the link, along with auth_enabled: false I also needed to add HTTP header X-Scope-OrgID as test [reference link] for error from loki: no org id

1 Like