Grafana -> Loki data lookup performance issue (something is slow)

[gerard1]

Hi All,

I’ve deployed a Loki setup behind an existing Grafana (TIG) stack. Everything works, or at least most of the time (see atttached images). The panels regularly show ‘no data’ while there definitely is. Not sure where to start looking/tweaking.

Accessing data with that same Grafana on Influx as the data source has no issues whatsoever and responds quick, even when gathering data from longer periods (months). It’s a simple nginx reverse proxy, with grafana as well as influx running as a native OS install (not docker).

Loki is currently setup/configured with a filesystem backend as storage (disks are SSD). I’m not sure if it’s inproper Loki config or filesystem storage being a bad choice.

Thanks a lot for any pointers.

Kind regards,

Gerard.

[gerard1]

Putting this in Loki config solved it:

query_range:
  split_queries_by_interval: 0
  parallelise_shardable_queries: false

querier:
  max_concurrent: 2048

frontend:
  max_outstanding_per_tenant: 4096
  compress_responses: true```

ref: https://github.com/grafana/loki/issues/5123#issuecomment-1025488801