Hi,
We have implemented loki using loki-distributed and for past few days, loki (added as data source in grafana) is unable to load labels. We use gcs as backend and I see below error messages in frontend pods. Can you please let me know what could be issue?
thanks,
Raghu
level=info ts=2023-11-15T23:36:03.511763903Z caller=frontend.go:280 msg=“received shutdown notification from querier” querier=grafana-loki-loki-distributed-querier-7
level=info ts=2023-11-15T23:37:04.985208688Z caller=frontend.go:280 msg=“received shutdown notification from querier” querier=grafana-loki-loki-distributed-querier-6
level=info ts=2023-11-15T23:38:04.610327439Z caller=frontend.go:280 msg=“received shutdown notification from querier” querier=grafana-loki-loki-distributed-querier-5
level=info ts=2023-11-15T23:39:09.445522768Z caller=frontend.go:280 msg=“received shutdown notification from querier” querier=grafana-loki-loki-distributed-querier-4
level=info ts=2023-11-15T23:40:05.566319555Z caller=frontend.go:280 msg=“received shutdown notification from querier” querier=grafana-loki-loki-distributed-querier-3
level=info ts=2023-11-15T23:41:18.878301596Z caller=frontend.go:280 msg=“received shutdown notification from querier” querier=grafana-loki-loki-distributed-querier-2
level=info ts=2023-11-15T23:42:01.351053479Z caller=frontend.go:280 msg=“received shutdown notification from querier” querier=grafana-loki-loki-distributed-querier-1
level=info ts=2023-11-15T23:42:55.236816195Z caller=frontend.go:280 msg=“received shutdown notification from querier” querier=grafana-loki-loki-distributed-querier-0
level=error ts=2023-11-15T23:53:11.781508901Z caller=retry.go:73 org_id=fake msg=“error processing request” try=0 err=“context canceled”
level=error ts=2023-11-15T23:53:11.781730988Z caller=stats.go:57 component=frontend msg=“failed to record query metrics” err="expected one of the *LokiRequest, *LokiInstantRequest, *LokiSeriesRequest, *LokiLabelNamesRequest, got "