Hi,
we are receiving this error:
Error: Request failed with status code 503
Get “http://loki-distributed-query-frontend.network-observability.svc.cluster.local:3100/loki/api/v1/query_range?query={app=\“netobserv-flowcollector\”,FlowDirection=\“0\”,SrcK8S_Namespace=\“network-observability\”}|~`SrcK8S_Type\“:\“Pod\”`|~`SrcK8S_Name\”:\“loki-distributed-query-frontend-86d879987-p8xms\”`&start=1664558260&limit=100”: context deadline exceeded (Client.Timeout exceeded while awaiting headers)
these are the logs from the frontend-querier when this happens:
level=info ts=2022-09-30T17:29:19.145299741Z caller=handler.go:174 org_id=fake msg=“slow query detected” method=GET host=loki-distributed-query-frontend.network-observability.svc.cluster.local:3100 path=/loki/api/v1/query_range time_taken=9.930590743s param_query=“{app="netobserv-flowcollector",FlowDirection="0",DstK8S_Namespace="network-observability"}|~DstK8S_Type\":\"Pod\"|~DstK8S_Name\":\"loki-distributed-query-frontend-86d879987-p8xms\"” param_start=1664558649 param_limit=100
level=info ts=2022-09-30T17:29:19.145779584Z caller=metrics.go:133 component=frontend org_id=fake latency=fast query=“{app="netobserv-flowcollector",FlowDirection="0",DstK8S_Namespace="network-observability"}|~DstK8S_Type\":\"Pod\"|~DstK8S_Name\":\"loki-distributed-query-frontend-86d879987-p8xms\"” query_type=filter range_type=range length=5m0.213825903s step=1s duration=9.930151177s status=200 limit=100 returned_lines=0 throughput=8.6MB total_bytes=85MB total_entries=13 queue_time=0s subqueries=6
level=error ts=2022-09-30T17:29:19.20950156Z caller=retry.go:73 org_id=fake msg=“error processing request” try=0 err=“context canceled”
level=error ts=2022-09-30T17:29:19.209560846Z caller=stats.go:57 component=frontend msg=“failed to record query metrics” err="expected one of the *LokiRequest, *LokiInstantRequest, *LokiSeriesRequest, *LokiLabelNamesRequest, got "
How can i fix this issue?