Grafana encountered an error while connecting to a remote Loki data source: Unable to fetch labels from Loki (Not Found)

Grafana Loki
,
1

My question background:

  1. grafana version = 9.5.3
  2. loki = 2.9.6, helm charts = loki-distributed-0.78.5

I want to integrate multiple data sources into the same Grafana for display.

The local loki cluster has been successfully connected and logs can be obtained. The address is:

http://loki-distributed-query-frontend.loki:3100

When connecting to a remote Loki, due to network limitations, I can only use ingress nginx as a proxy,My configuration for remote ingress nginx is as follows:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/rewrite-target: /$2
  name: http-loki-myapiserver
  namespace: myapiserver
spec:
  rules:
  - host: myapiserver.loki-local.com
    http:
      paths:
      - backend:
          service:
            name: loki-apiserver
            port:
              number: 3100
        path: /loki(/|$)(.*)
        pathType: Prefix

Grafana configuring remote data sources：

http://myapiserver.loki-local.com/loki

Error reported in save&test：

Unable to fetch labels from Loki (Not Found), please check the server logs for more details

Grafana logs：

logger=context userId=1 orgId=1 uname=admin t=2024-05-23T03:45:37.315914664Z level=info msg="Request Completed" method=GET path=/api/datasources/uid/e13e1e70-6fd3-493c-959f-4fb91380949e/resources/labels status=404 remote_addr=2xx.xx.xx.x8 time_ms=111 duration=111.128925ms size=23 referer=http://1x.xx.xx.x5:31658/connections/your-connections/datasources/edit/e13e1e70-6fd3-493c-959f-4fb91380949e handler=/api/datasources/uid/:uid/resources/*

Remote ingress nginx logs：

 - - [23/May/2024:06:16:59 +0000] "GET /loki/loki/api/v1/labels?start=1716444418990000000&end=1716445018990000000 HTTP/1.1" 404 9 "-" "Grafana/9.5.3" 708 0.000 [-] [] - - - - d2fd120e1a7022d2b6c74e0984d167ca

I have already done a rewrite operation in my ingress, why hasn’t it taken effect?
i hope to receive help, thank you.

2

Does your remote Loki have one extra /loki in the URL path? Your error message says /loki/loki/api/v1/labels, looks a bit weird.

3

The first /loki is the path added by my ingress nginx,
The second /loki is built-in when Grafana requests Loki.

I have checked the complete path of Grafana’s request for Loki, which is:

/loki/api/v1/labels

Add /loki in my forwarding and rewriting rules,I think this is reasonable.

and it is normal for me to access it in the following way:

curl xxxxx/loki/loki/api/v1/labels
{"status":"success","data":["app","component","container","filename","instance","namespace","node_name","pod","type"]}
4

Your nginx log is showing 404, so I’d start there. Also perhaps share your Nginx configuration.

5

I am using kubernetes/ingress-nginx,Here is my configuration:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/rewrite-target: /$2
  name: http-loki-myapiserver
  namespace: myapiserver
spec:
  rules:
  - host: myapiserver.loki-local.com
    http:
      paths:
      - backend:
          service:
            name: loki-apiserver
            port:
              number: 3100
        path: /loki(/|$)(.*)
        pathType: Prefix
6

  1. Is http://loki-apiserver:3100 a valid host? Are you able to perform Loki API against this host from either your kubernetes host or somewhere internal?

  2. Check your Loki writer or reader logs and see if there is anything obvious there. If the path doesn’t match or something of that sort (which would be 404) you’d usually see something in the logs.