I have 1 query frontend deployed on a vm with 2 queriers each on a separate vm pulling queries from the frontend. Queries are executed in grafana.
When executing queries (refreshing a dashboard) in a small time-range everything works fine, however at around a 12 hours time-range after a while i’m getting a 502 bad gateway.
Values I tried to edit:
timeout = 600
I’m not sure if this is a grafana or loki problem, but I do think it’s grafana related since the 502 isn’t sent from the query-frontend. The querier-frontend gets a tcp write error.
I was able to solve it, it was a timeout on both ends sometimes Grafana and sometimes Loki,depending on the configurations I changed.
I never ran the configurations at both Grafana and Loki always seeing no affect made me think the configuration is useless, turns out you need all of them.
I have the same issue, how did you resolve this? Thanks.
As I mentioned, look at the grafana configuration:
In grafana 8 you can set the timeout from the UI when configuring the data source.
From Loki i’m not exactly sure what fixed this, look into the following querier configuration:
Thank you @m3r1 , it turns out my issue is not just timeout issue, I have performance issues on loki server, I posted my issue here loki crashed for large queries · Issue #4582 · grafana/loki · GitHub.
what worked for me:
- name: Loki