Grafana Loki - 504 gateway error in explore logs

Grafana Loki
1

Hi All,

We are facing 504 gateway error in Grafana explore->logs. What could be the issue? Dont see any errors in the gateway and verified the timeouts as well. Please advise what is causing this and how to fix it

2

Need a bit more details. Couple of questions:

  1. Was it working before? If so, what changed?
  2. Are you running Loki in simple scalable mode or micro service mode?
  3. Does the write path work, or does the write path also give you 504?
  4. Does it work if you send an API call to the gateway?
  5. Does it work if you send an API call directly to the querier, bypassing gateway?
  6. Check other components for error logs.
3
  1. We didn’t make any changes w.r.t gateway, we started facing this error with 24-48 hr queries with more data results
  2. We are running microservices mode
  3. yes, logs are getting pushed from fluentd to loki
  4. Not all calls are failing
  5. noticed some chunk memory issues, have increased it to 8Gi now.
4

How much logs do you have for, say, 24 hours?

If you are running into performance issue with queries that cover a long period of time, you’ll want to:

  1. Make sure you are using query splitting.
  2. Scale up your queries. Usually it’s better to have more queries as opposed to fewer bit bigger queriers.
5
  1. split_query_intrval is set to 1h, max_query_parallelism is 48.
  2. using 12 queriers as per your suggestion in the other topic 4gb/2cpu