Hello, I’m having a problem trying to add my Loki as a datasource in my Grafana, both located in the same EKS cluster. I even did a kubectl exec -it grafana- -n monitoring – curl http://loki:3100 test and the test shows that both communicate normally.
This probably isn’t answerable without many more details of your setup - but you’re not trying to connect to http://loki:3100 you’re connecting to http://loki.monitoring:3100.
In these kinds of scenarios, 9 times out of 10, the issue isn’t anything to do with Grafana or Loki, but it has to do with DNS resolution and how it might be different on the grafana machine vs. where you’re testing it.
So the solution tends to involve diving into the EKS configuration to figure out what’s what. But the first step would be to try kubectl exec -it grafana- -n monitoring – curl http://loki.monitoring:3100
Hello Davi. All good? Thanks for answering! Below is an image of the log I obtained after running the command.
So now it makes sense why your loki connection is failing, because there’s no page there.
I can’t help with why that URL isn’t resolving, it depends too much on how your EKS deployment is setup. But you previously checked that http://loki:3100 worked, so try that. If that doesn’t work in your config, then I’m afraid you’ve got an EKS configuration issue, and you should post a different question with much more info about how loki & grafana are deployed.
Hi Davi, I got the same error.
I`m deploying the loki-distributed mode in a Openshif cluster
The component loki-loki-distributed-distributor is displaying this on the logs
What can I do?
For whoever stumble upon this issue, updating loki image to version 2.9.3 resolves mine.
Thank to romosa
Hey Davi, I am facing the same error in on-prem helm deployment could you please help me.
Hiya folks – this error is rather generic and be caused by a lot of different factors. I’m going to close this thread as a result, but with a summary of what we’re discussing here, and some pointers on what to check out.
You can open an additional question but please provide more information and please try these items first.
Summary: The thread discusses issues with connecting Grafana to Loki as a data source, primarily focusing on DNS resolution and service accessibility within Kubernetes and OpenShift environments. People face issues due to incorrect service URL configuration, and it’s really important to check component logs for errors and report what they say so others can help. @phamthanhalv suggested updating the Loki image to version 2.9.3, which resolved their issue.
Recommendations if you’re having this problem
- Ensure correct DNS resolution for Loki’s service URL in your Grafana configuration.
- Verify that the Loki service is accessible from the Grafana pod using the correct namespace and service name.
- Consider updating the Loki image to the latest stable version if encountering persistent issues.
- Check the logs of Loki components for any errors or misconfigurations.
- If using multi-tenant mode, ensure the
X-Scope-OrgIDheader is correctly configured in Grafana. - For OpenShift users, ensure that the Loki distributed mode is correctly set up and all components are functioning.
Other resources which might help: