Connecting Grafana to Loki from a subpath

Grafana Loki
1

I’m having an issue connecting Grafana to a Loki instance running under a subpath.

I can successfully connect to the Loki instance through my browser, and both the /ready and loki/api/v1/labels endpoints return valid values when running under the subpath. I can even connect a promtail instance to Loki through this subpath.

However, when adding the Loki datasource to Grafana, I receive a 404 error on the health checkpoint connection, and the connection fails. Running Loki without the subpath allows me to connect Grafana.

I’m running grafana and loki in rootless podman containers.

Here is my Loki configuration:

auth_enabled: false

server:
  http_listen_port: 3100
  http_path_prefix: /logs

And here are the snippets from the Grafana logs:

logger=tsdb.loki endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=checkHealth pluginId=loki dsName=Loki dsUID=1mtP_JAVz uname=USERNAME fromAlert=false t=2024-07-12T13:52:00.770811639Z level=info msg="Prepared request to Loki" duration=23.053µs queriesLength=1 stage=prepareRequest runInParallel=false

logger=tsdb.loki endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=checkHealth pluginId=loki dsName=Loki dsUID=1mtP_JAVz uname=USERNAME fromAlert=false t=2024-07-12T13:52:00.773442686Z level=error msg="Error received from Loki" duration=2.370384ms stage=databaseRequest statusCode=404 contentLength=19 start=1970-01-01T00:00:01Z end=1970-01-01T00:00:04Z step=1s query=vector(1)+vector(1) queryType=instant direction=backward maxLines=0 supportingQueryType=none lokiHost=loki:3100 lokiPath=/logs/loki/api/v1/query status=error error="404 page not found\n" statusSource=downstream

logger=tsdb.loki endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth t=2024-07-12T13:52:00.77358738Z level=error msg="Loki health check failed" error="error from loki: 404 page not found\n"

And here are the Loki debug logs when I try to add the Grafana datasource:

level=info ts=2024-07-12T13:54:48.657045311Z caller=roundtrip.go:309 org_id=fake traceID=42d08784b867c51d msg="executing query" type=instant query=vector(1)+vector(1) query_hash=2536976802

level=debug ts=2024-07-12T13:54:48.657222862Z caller=split_by_range.go:72 middleware=InstantQuery.splitByRangeVector org_id=fake traceID=42d08784b867c51d msg="mapped instant query" interval=30m0s noop=true original=vector(1)+vector(1) mapped="(vector(1.000000) + vector(1.000000))"

ts=2024-07-12T13:54:48.65741913Z caller=spanlogger.go:86 middleware=QueryShard.astMapperware org_id=fake traceID=42d08784b867c51d org_id=fake traceID=42d08784b867c51d level=debug no-op=true mapped="(vector(1.000000) + vector(1.000000))"

level=debug ts=2024-07-12T13:54:48.658083122Z caller=logging.go:118 traceID=42d08784b867c51d orgID=fake msg="GET /logs/loki/api/v1/query?direction=backward&query=vector%281%29%2Bvector%281%29&time=4000000000 (404) 1.195687ms"

Removing the http_path_prefix: /logs and configuring Grafana to connect on the root path makes it work.

I’d appreciate any help in figuring out this issue!

2

I’ve never used subpath, so probably not much help. However, I’d look at your Grafana query inspector and look at the API calls being performed, and see if there is anything weird there.

Also make sure API works with subpath as well just to rule it out.