Small Loki queries slows Grafana to a crawl

Grafana
,
1

Hey folks!

I have successfully deployed Loki (scalable mode Helm chart, no value modifications) on my K8S and connected it to the Grafana instance that comes with kube-prometheus-stack. However, whenever I try to query the logs, Grafana becomes unresponsive to all HTTP requests and eventually returns “Load failed” even though I collected close to no logs yet. I am querying with the 1000 logs limit in place. This must be a misconfiguration on my part! I’d appreciate any help in getting things running. Here’s my values file for kube-prometheus-stack:

grafana:
  additionalDataSources:
    - name: otel-loki
      type: loki
      default: true
      uid: some-loki-source
      access: proxy
      url: http://loki-gateway.observability.svc.cluster.local
      jsonData:
        httpHeaderName1: X-Scope-OrgID
        maxLines: 1000
      secureJsonData:
        httpHeaderValue1: MyOrg
2

Use standard approach: increase Grafana log level and watch Grafana server logs to see what’s happening on Grafana server/pod.