Unable to fetch labels from Loki (Failed to call resource), please check the server logs for more details

Grafana Loki
, , ,
1

  • What Grafana version and what operating system are you using?
    v9.3.6

  • What are you trying to achieve?
    connect to loki data source from grafana

  • How are you trying to achieve it?
    installed components through helm chart

  • What happened?
    installed helm, grafana, promtail through helm charts and all are running successfully but I was unable to connect to loki data source from grafana

  • What did you expect to happen?
    get the logs in grafana dashboard

  • Can you copy/paste the configuration(s) that you are having problems with?

  • Did you receive any errors in the Grafana UI or in related logs? If so, please tell us exactly what they were.

  • Did you follow any online instructions? If so, what is the URL?

I checked all pods are up and running, loki is publishing metrics on
http://loki-backend:3100/metrics

but when I try to add the loki in grafana like this way http://loki-backend:3100 it is not connecting

here is my loki configmap value

----
auth_enabled: false
common:
  path_prefix: /var/loki
  replication_factor: 3
  storage:
    s3:
      access_key_id: enterprise-logs
      bucketnames: chunks
      endpoint: loki-minio.oal-loki-poc.svc:9000
      insecure: true
      s3forcepathstyle: true
      secret_access_key: supersecret
limits_config:
  enforce_metric_name: false
  max_cache_freshness_per_query: 10m
  reject_old_samples: true
  reject_old_samples_max_age: 168h
  split_queries_by_interval: 15m
memberlist:
  join_members:
  - loki-memberlist
query_range:
  align_queries_with_step: true
ruler:
  storage:
    s3:
      bucketnames: ruler
    type: s3
runtime_config:
  file: /etc/loki/runtime-config/runtime-config.yaml
schema_config:
  configs:
  - from: "2022-01-11"
    index:
      period: 24h
      prefix: loki_index_
    object_store: s3
    schema: v12
    store: boltdb-shipper
server:
  grpc_listen_port: 9095
  http_listen_port: 3100
storage_config:
  hedging:
    at: 250ms
    max_per_second: 20
    up_to: 3
table_manager:
  retention_deletes_enabled: false
  retention_period: 0
2

I intentionally removed “p” in http because site not allowing more than 2 links in a post
observation:
when I put the this in url (htt://loki-backend:3100 or htt://loki-gateway) I get message in log like this (error="404 page not found)

" logger=context userId=1 orgId=1 uname=admin t=2023-02-10T15:51:50.622326971Z level=error msg="Failed to call resource" error="404 page not found\n" traceID=

logger=context userId=1 orgId=1 uname=admin t=2023-02-10T15:51:50.622425027Z level=error msg="Request Completed" method=GET path=/api/datasources/3/resources/labels status=500 remote_addr=[::1] time_ms=6 duration=6.122883ms size=51 referer=htt://localhost:3001/datasources/edit/mYX3no0Vz handler=/api/datasources/:id/resources/"

when I put this in URL (htt://localhost:3100) I know the localhost is not right one in my case but for demo purpose I’m doing this (Get "htt://localhost:3100/loki/api/v1/labels?)

logger=context userId=1 orgId=1 uname=admin t=2023-02-10T15:53:20.514537443Z level=error msg="Failed to call resource" error="Get \"htt://localhost:3100/loki/api/v1/labels?start=1676043800268000000&end=1676044400268000000\": dial tcp [::1]:3100: connect: connection refused" traceID=
logger=context userId=1 orgId=1 uname=admin t=2023-02-10T15:53:20.514631089Z level=error msg="Request Completed" method=GET path=/api/datasources/3/resources/labels status=500 remote_addr=[::1] time_ms=3 duration=3.933432ms size=51 referer=htt://localhost:3001/datasources/edit/mYX3no0Vz handler=/api/datasources/:id/resources/
3

found the solution:
during the helm install it installed the chart with version 4.4.2 which installed lot of components and loki-backend but after removing 4.4.2 version and installed 2.16.0 chart version it only installed loki. This version worked fine and able to connect to loki data source from grafana