Grafana can't show more than 3 loki panels in a dashbaord

I am trying to use loki and grafana for generating and visualising a number of metrics based on logs. When I add a 4th panel to my dashboard all panels start failing when trying to query loki for durations longer than a couple of hours with the error: “Too many outstanding requests”.

I am using a simple monolith deployment (target=all) of loki with the following configuration added as mentioned in some other issue threads similar to mine:

limits_config:
  split_queries_by_interval: 24h

frontend:
  max_outstanding_per_tenant: 2048

But these settings did not help in a noticeable way.
I also tried deploying a separate query-frontend as the documentation did not make it clear if the monolith deployment included the query frontend or not but that seemed to have made the issue worse.

I was not able to find any useful logs from either loki or grafana, grafana seems to be receiving response code 400 from loki and loki logs state that the query fails because the context has been cancelled. There are no proxies between loki and grafana and they are running on the same machine. Dump logs are quite hard to look through as I don’t really know what to look for.

Is there anything I can try to figure out why this is happening or fix the issue?
Also the documentation for configuring loki seems to be quite lacking, and the little documentation there is is mostly geared towards kubernetes.

2 Likes

Same issue. Tough to find guidance.

Forgot about this post but I eventually found what parameters to tweak, adding the following lines fixed it for me:

limits_config:
    split_queries_by_interval: 24h  # default value is too low and causes queries for long periods to hang and eventually be rejected
query_scheduler:
    max_outstanding_requests_per_tenant: 2048  # default value is too low and limits number of concurrent queries and therefore dashboards

Yeah, Loki documentation is quite lacking. especially when it comes to setting up a monolith deployment and tuning it :frowning: I do feel like it’s gotten slightly better over the last year or so.