I am trying to use loki and grafana for generating and visualising a number of metrics based on logs. When I add a 4th panel to my dashboard all panels start failing when trying to query loki for durations longer than a couple of hours with the error: “Too many outstanding requests”.
I am using a simple monolith deployment (target=all) of loki with the following configuration added as mentioned in some other issue threads similar to mine:
limits_config: split_queries_by_interval: 24h frontend: max_outstanding_per_tenant: 2048
But these settings did not help in a noticeable way.
I also tried deploying a separate query-frontend as the documentation did not make it clear if the monolith deployment included the query frontend or not but that seemed to have made the issue worse.
I was not able to find any useful logs from either loki or grafana, grafana seems to be receiving response code 400 from loki and loki logs state that the query fails because the context has been cancelled. There are no proxies between loki and grafana and they are running on the same machine. Dump logs are quite hard to look through as I don’t really know what to look for.
Is there anything I can try to figure out why this is happening or fix the issue?
Also the documentation for configuring loki seems to be quite lacking, and the little documentation there is is mostly geared towards kubernetes.