Unable to fetch labels from Loki (Failed to call resource)

Grafana Loki
#1

We are working on PLG (promtail , loki and grafana) stack setup , loki is deployed in distributed microservice mode . Promtail and loki is working as expected but when trying to add loki as a datasource in grafana dashboard we are getting the below error (PFA screenshot for the same). Can you please help on this.

Unable to fetch labels from Loki (Failed to call resource)

Error logs :
logger=sqlstore t=2022-07-07T07:47:38.451264825Z level=info msg=“Database locked, sleeping then retrying” error=“database is locked” retry=0
logger=context traceID=00000000000000000000000000000000 userId=1 orgId=1 uname=admin t=2022-07-07T08:14:24.575931983Z level=info msg=“Request Completed” method=GET path=/api/live/ws status=0 time_ms=1 duration=1.072052ms size=0 referer= traceID=00000000000000000000000000000000
logger=context traceID=00000000000000000000000000000000 userId=1 orgId=1 uname=admin t=2022-07-07T08:14:32.576690448Z level=error msg=“Failed to call resource” error=“\r\n404 Not Found\r\n\r\n

404 Not Found<
logger=context traceID=00000000000000000000000000000000 userId=1 orgId=1 uname=admin t=2022-07-07T07:40:14.056975972Z level=info msg=“Request Completed” method=GET path=/api/live/ws status=0 time_ms=1 duration=1.176239ms size=0 referer= traceID=00000000000000000000000000000000
logger=context traceID=00000000000000000000000000000000 userId=1 orgId=1 uname=admin t=2022-07-07T07:40:26.484892541Z level=error msg=“Failed to call resource” error=”\r\n404 Not Found\r\n\r\n

404 Not Found

\r\n
nginx\r\n\r\n\r\n" traceID=00000000000000000000000000000000
logger=context traceID=00000000000000000000000000000000 userId=1 orgId=1 uname=admin t=2022-07-07T07:40:26.484957144Z level=error msg=“Request Completed” method=GET path=/api/datasources/1/resources/labels status=500 time_ms=16 duration=16.982728ms size=83

#2

Could you please format this nicely. cant sort out what is what

#3

@yosiasz

Please find below latest logs in proper format

logger=context traceID=00000000000000000000000000000000 userId=1 orgId=1 uname=admin t=2022-07-18T12:13:58.745370165Z level=error msg="Failed to call resource" error="Get \"http://loki.ingress.eval.lp.shoot.live.k8s-hana.ondemand.com:3100/api/datasources/1/resources/labels/loki/api/v1/labels?start=1658145828524000000&end=1658146428524000000\": dial tcp 18.196.216.18:3100: i/o timeout" traceID=00000000000000000000000000000000
logger=context traceID=00000000000000000000000000000000 userId=1 orgId=1 uname=admin t=2022-07-18T12:13:58.745463247Z level=error msg="Request Completed" method=GET path=/api/datasources/1/resources/labels status=500 remote_addr=165.1.238.31 time_ms=10003 duration=10.003021789s size=83 referer=https://grafana.ingress.eval.lp.shoot.live.k8s-hana.ondemand.com/datasources/edit/P982945308D3682D1 traceID=00000000000000000000000000000000

Please consider these logs to investigate. Also let me know if you need any other details. We tried a lot of things as per document but still no luck. Not sure what we are missing.