Hello!
We are seeing increased number of alerts:
Error: failed to execute query A: Get "https://prometheus-server.local/api/v1/query_range?end=1693311240********************************": net/http: request canceled (Client.Timeout exceeded while awaiting headers)
and
Error: failed to execute query A: Get "https://prometheus-server.local/api/v1/query_range?end=1693311240&query=changes%28script_running%7Bhost%3D%27************************************************": net/http: timeout awaiting response headers
after we upgraded from grafana 9.1.4 to 10.0.3
We’ve seen similar errors sporadicaly with 9.1.4 but it’s far more frequent since we upgraded to 10.0.3
We tried increasing dataproxy timeout to 60s and it’s pretty much the same, only it timeouts after 60s instead of 30.
We are running Nginx reverse proxy in front of prometheus datasource but we are able to see in logs that Ngxinx was able to respond with 200 before the triggered alert.
Does anyone know where to look for additional header response settings or something like that? It looks like something has change in recent versions and we would like to sort this.
Thank you!