Unable to connect with Loki - ssl handshake failed

Grafana
, ,
1

I’m running into an issue accessing my Loki instance deployed on Azure Kubernetes Service (AKS). I’m using the Nginx Ingress controller to expose Loki externally, and Promtail is running within the cluster to ship logs.

Setup:

  • Platform: AKS

  • Service: Loki (standard stack, deployed via Helm/YAML)

  • Log Shipper: Promtail

  • Ingress Controller: Nginx Ingress

  • Ingress Config: No Auth

  • Domain: example.org (example, using my actual domain)

Problem:

My Ingress configuration seems partially correct. I have configured it to route traffic based on a specific path prefix:

  • :white_check_mark: I can successfully access https://example.org/rewardsy-loki/ready (returns 200 OK after Basic Auth).

  • :white_check_mark: I can successfully access https://example.org/rewardsy-loki/metrics (returns Loki metrics after Basic Auth).

  • :cross_mark: Accessing https://example.org/ returns a 404 (This is somewhat expected as it doesn’t match my specific Ingress path rule).

  • :cross_mark: Accessing https://example.org/rewardsy-loki/ (the base path defined in the Ingress) also returns a 404 . This 404 seems to be coming from the Loki service itself after the Ingress routing and path rewrite.

  • :cross_mark: When trying to add Loki as a data source in Grafana using the URL https://example.org/rewardsy-loki (and providing the correct Basic Auth credentials configured in Grafana), I get the error: “Unable to connect with Loki. Please check the server logs for more details.” or sometimes a generic HTTP Error/Network Error.

Ingress Configuration:

Here’s my current Ingress resource YAML:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: rewardsy-loki-ingress
  annotations:
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/use-regex: "true"
    nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
  ingressClassName: nginx
  rules:
    - http:
        paths:
          - path: /rewardsy-loki(/|$)(.*)
            pathType: Prefix
            backend:
              service:
                name: loki-stack
                port:
                  number: 3100

Logs :

  • [13/Apr/2025:10:50:42 +0000] "GET /rewardsy-loki/loki/api/v1/query?direction=backward&query=vector%281%29%2Bvector%281%29&time=4000000000 HTTP/1.1" 400 65 "-" "Grafana/10.4.15 AzureManagedGrafana/latest" 397 0.001 [loki-stack-loki-stack-3100] [] 10.244.5.47:3100 65 0.000 400 fecf5f34b97a88252b20fe8608bdf1f8

image
image660×828 48 KB

  • I have verified the logs in the ingress-controller. It was saying this SSL_do_handshake() failed (SSL: error:141CF06C:SSL routines:tls_parse_ctos_key_share:bad key share) while SSL handshaking

But I dont have any SSL configured

  • I tried to check the logs further and it was of no use.
2

That’s weird. Your test examples are using https and ingress has annotations to redirect traffic to https.

So it doesn’t make sense to use http in the datasource config. Try https.

3

My apologies,

I have tried both options and neither worked (But from browser, both links are working)

image
image645×187 21.9 KB

image
image645×187 20.8 KB

Could it be an ingress issue,how to verify?

4

You have a lot of inconsistencies:

  • is SSL enabled or not - what is doing SSL and how - because you proved that https AND HTTP works for you
  • Is auth enabled or not - You claim that there is no auth, but you claim that you are sending basic auth
  • you have an error when Loki responded with response code 400 and that there is an SSL error - there is no way for that - it looks like you are mixing unrelated logs

Please provide reproducible examples and issue-related logs. You have a lot of moving parts, which can be disconfigured.