when i query data from grafana for 1 hour its timeout after 30s and getting this kind of error:
Status: 504. Message: Get “http://lab.example.com/loki/api/v1/query_range?direction=backward&end=1730897453861000000&query=sum(count_over_time(stg-app-notification&start=1730893852000000000&step=2000ms”: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
Also i’ve promtail & Loki setup by using offical grafana loki helm chart. From grafana, when I am running a query to loki, its giving me 504 & the read pods is giving this error -
level=error ts=2024-11-06T12:51:24.176599593Z caller=scheduler_processor.go:158 org_id=fake msg="error notifying scheduler about finished query" err=EOF addr=10.14.38.35:9095
level=error ts=2024-11-06T12:51:24.176059548Z caller=scheduler_processor.go:106 msg="error processing requests from scheduler" err="rpc error: code = Canceled desc = context canceled" addr=10.14.37.17:9095
level=error ts=2024-11-06T12:51:24.176135338Z caller=scheduler_processor.go:208 org_id=fake frontend=10.14.38.15:9095 msg="error notifying frontend about finished query" err="rpc error: code = Canceled desc = context canceled"
level=error ts=2024-11-06T12:51:24.17619066Z caller=scheduler_processor.go:252 org_id=fake frontend=10.14.38.15:9095 msg="error health checking" err="rpc error: code = Canceled desc = context canceled"
level=error ts=2024-11-06T12:51:24.17622659Z caller=scheduler_processor.go:158 org_id=fake msg="error notifying scheduler about finished query" err=EOF addr=10.14.37.19:9095
level=error ts=2024-11-06T12:51:24.176400204Z caller=scheduler_processor.go:208 org_id=fake frontend=10.14.38.51:9095 msg="error notifying frontend about finished query" err="rpc error: code = Unavailable desc= error reading from server: read tcp 10.14.37.21:37242->10.14.38.15:9095: use of closed network connection"
level=error ts=2024-11-06T12:51:24.176458684Z caller=scheduler_processor.go:252 org_id=fake frontend=10.14.38.15:9095 msg="error health checking" err="rpc error: code = Canceled desc = grpc: the client connection is closing"
level=error ts=2024-11-06T12:51:24.176497415Z caller=scheduler_processor.go:208 org_id=fake frontend=10.14.38.15:9095 msg="error notifying frontend about finished query" err="rpc error: code = Unavailable desc= error reading from server: read tcp 10.14.37.21:37242->10.14.38.15:9095: use of closed network connection"
level=warn ts=2024-11-06T12:51:24.176522506Z caller=pool.go:196 msg="removing frontend failing healthcheck" addr=10.14.38.15:9095 reason="rpc error: code = Unavailable desc = error reading from server: read tcp 10.14.37.21:37242->10.14.38.15:9095: use of closed network connection"
level=error ts=2024-11-06T12:51:24.176534396Z caller=scheduler_processor.go:252 org_id=fake frontend=10.14.38.15:9095 msg="error health checking" err="rpc error: code = Canceled desc = grpc: the client connection is closing"
Could anyone please give me insight to solve this so that i can able to cater more than 1 hour logs?