Log retrieval fails with "Bad Gateway"... Cannot poll logs + 1 hour | Help with Config

My setup is as follows:

  1. Servers ship logs to a centralized RSYSLOG box.
  2. Log Server ships logs to loki via promtail
  3. Loki Ingests loks and grafana is used to sort and filter them

I have a LOT of logs and each time I try to search / sort / query them I get:
but I can see logs if I just use {job=~".+"}|=“error” but only for a period < 1 hour. It seems loki is timing out during the requests.However it will retrieve results if a lesser time is selected.

Should I NOT aggregate my logs before shipping and create custom promtail configs for each of my servers? I have a requirement to ship all logs in every folder so using …//* or …//*.log is advisable?

Open up your browsers dev tools, and then the network tab. it should tell you what part of loki is giving you the bad gateway error.

I see there is a backwards limit based on the error url: