We are working on PLG (promtail , loki and grafana) stack setup , loki is deployed in distributed microservice mode . Promtail and loki is working as expected but when trying to add loki as a datasource in grafana dashboard we are getting the below error (PFA screenshot for the same). Can you please help on this.
Unable to fetch labels from Loki (Failed to call resource)
I ran into this exact issue today. I am using Promtail running locally on my windows computer together with Loki. Both Promtail and Loki were running with zero issues or errors but Grafana still showed “failed to call resource”.
What worked for me (and has worked once before is).
shut down Loki and Promtail
Delete the Loki configuration folder (for me /tmp/*) which deletes all chunks, index files and so on.
restart Loki and Promtail and wait for a while. Promtail needs to scrape and send to Loki, which is not the fastest due to my setup
reload localhost:3000. Remove the data source Loki and remake it.
It makes no sense to me but perhaps others will find this works for them too.
Same issue here. I upgraded grafana from 8.3.11 to 9.3.6 (where it worked fine) and started seeing this issue.
The only trace of this issue in logs is: logger=context userId=1 orgId=1 uname=admin t=2023-02-21T14:45:04.833177993Z level=error msg="Request Completed" method=GET path=/api/datasources/4/resources/labels status=500 remote_addr=126.96.36.199 time_ms=1 duration=1.809072ms size=51 referer=https://logs.theguru.co/datasources/edit/7Rn3jrJVk handler=/api/datasources/:id/resources/* which does not say much