# Request canceled (Client.Timeout exceeded while awaiting headers) **URL:** \ **Category:** Configuration\ **Created:** [July 16, 2024, 2:47pm UTC](https://community.grafana.com/t/request-canceled-client-timeout-exceeded-while-awaiting-headers/126348 "2024-07-16T14:47:18Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![delaramhamraz](https://avatars.discourse-cdn.com/v4/letter/d/57b2e6/32.png) [@delaramhamraz](https://community.grafana.com/u/delaramhamraz)\ **Post date:** [July 16, 2024, 2:47pm UTC](https://community.grafana.com/t/request-canceled-client-timeout-exceeded-while-awaiting-headers/126348/1 "2024-07-16T14:47:18Z")
#### Problem Description: I have a complex Prometheus query that runs successfully and quickly in Thanos but consistently times out in Grafana on big time range like 120 days. The query is as follows: ```auto avg(sum_over_time( (sum by(container)(kube_pod_container_status_running{namespace=~"keycloak|dev", container=~"keycloak|test-server|test-web"}) >= bool 1)[$__range:] ) / count_over_time( (sum by(container)(kube_pod_container_status_running{namespace=~"keycloak|dev", container=~"keycloak|test-server|test-web"}))[$__range:] ) * 100) ``` with this query I want to get the global availability of my applications. ### Observations: - The query works perfectly in Thanos without timeouts. - Grafana still times out despite increasing the timeout settings. - The memory usage of `thanos-storegateway` is high, but it’s managed within the increased resource limits. Why does the query time out in Grafana but not in Thanos? Are there other settings in Grafana or Thanos that I should adjust? or is there a way to simplify the query ? I would be really thankful if anyone have any ideas help me ---
**Author:** ![codi639](https://avatars.discourse-cdn.com/v4/letter/c/ba9def/32.png) [@codi639](https://community.grafana.com/u/codi639)\ **Post date:** [July 16, 2024, 3:22pm UTC](https://community.grafana.com/t/request-canceled-client-timeout-exceeded-while-awaiting-headers/126348/2 "2024-07-16T15:22:54Z")
Hello @delaramhamraz welcome to the forum, Did you try to increase the Grafana timeout in the `grafana.ini` or directly in the datasource settings? ---
**Author:** ![delaramhamraz](https://avatars.discourse-cdn.com/v4/letter/d/57b2e6/32.png) [@delaramhamraz](https://community.grafana.com/u/delaramhamraz)\ **Post date:** [July 16, 2024, 3:36pm UTC](https://community.grafana.com/t/request-canceled-client-timeout-exceeded-while-awaiting-headers/126348/3 "2024-07-16T15:36:14Z")
Hello 🙂 Thank you yes I did. but i have the same problem. although if the timeout did work, I would still have the problem of waiting too long for the data to be loaded . since i want to present this dashboard to other people, I can’t have them wait that long ---
**Author:** ![codi639](https://avatars.discourse-cdn.com/v4/letter/c/ba9def/32.png) [@codi639](https://community.grafana.com/u/codi639)\ **Post date:** [July 16, 2024, 8:59pm UTC](https://community.grafana.com/t/request-canceled-client-timeout-exceeded-while-awaiting-headers/126348/4 "2024-07-16T20:59:16Z")
Yeah it make sense haha. I don’t really have a solution right now but some ideas… First of all, but I assume you got this, make sure your servers (are you using multiple server?) have sufficient ressources (RAM or core for example). Your queries looks very complex so, I’d say, try to find a way to simplify them at maximum. I don’t know if that’s possible, but maybe you can try to make your Thanos service handle big queries in a different way, which would help. In the same idea, you can try to build multiple table, which would be filled by the result of some intermediate queries. Then build a simplified query? Those are only ideas. I’ll take a better look in two days, but I can’t promise a solution! Best regards ---
**Author:** ![jangaraj](https://sea2.discourse-cdn.com/grafana/user_avatar/community.grafana.com/jangaraj/32/39197_2.png) [@jangaraj](https://community.grafana.com/u/jangaraj)\ **Post date:** [July 16, 2024, 9:21pm UTC](https://community.grafana.com/t/request-canceled-client-timeout-exceeded-while-awaiting-headers/126348/5 "2024-07-16T21:21:17Z")
I would optimise default query options, e.g.: ![obrázok](https://us1.discourse-cdn.com/grafana/original/3X/f/d/fd2211f3e4341481d071db5f52e5842d5eb24ad9.png) ---
**Author:** ![delaramhamraz](https://avatars.discourse-cdn.com/v4/letter/d/57b2e6/32.png) [@delaramhamraz](https://community.grafana.com/u/delaramhamraz)\ **Post date:** [July 17, 2024, 10:24am UTC](https://community.grafana.com/t/request-canceled-client-timeout-exceeded-while-awaiting-headers/126348/6 "2024-07-17T10:24:28Z")
Thanks for the ideas, I would really much like to simplify the query, but i don’t know how really. i just want to show the global availability of the applications, and it wasn’t me who wrote the query. so i’m not sure if i understand really the query ---
**Author:** ![delaramhamraz](https://avatars.discourse-cdn.com/v4/letter/d/57b2e6/32.png) [@delaramhamraz](https://community.grafana.com/u/delaramhamraz)\ **Post date:** [July 17, 2024, 10:26am UTC](https://community.grafana.com/t/request-canceled-client-timeout-exceeded-while-awaiting-headers/126348/7 "2024-07-17T10:26:25Z")
thanks for the response. I tried with min interval 1 day but i still have the same problem. so just for trying i put it at 5d and it worked, but doesn’t that mean that i am losing a lot of data points? and the percentage that i am showing on the graph Gauge, is untrustworthy ? ---
**Author:** ![jangaraj](https://sea2.discourse-cdn.com/grafana/user_avatar/community.grafana.com/jangaraj/32/39197_2.png) [@jangaraj](https://community.grafana.com/u/jangaraj)\ **Post date:** [July 17, 2024, 10:30am UTC](https://community.grafana.com/t/request-canceled-client-timeout-exceeded-while-awaiting-headers/126348/8 "2024-07-17T10:30:54Z")
That doesn’t sound very good: create timeseries with expensive query, when you just need single number for gauge panel. I would say you can use even longer period there. In theory 120d. ---
**Author:** ![delaramhamraz](https://avatars.discourse-cdn.com/v4/letter/d/57b2e6/32.png) [@delaramhamraz](https://community.grafana.com/u/delaramhamraz)\ **Post date:** [July 17, 2024, 10:43am UTC](https://community.grafana.com/t/request-canceled-client-timeout-exceeded-while-awaiting-headers/126348/9 "2024-07-17T10:43:42Z")
Do you think it’s necessary to divide the sum of my running pods by the number of times that prometheus searched for the metrics ? i can’t quite understand the division here ---
**Author:** ![jangaraj](https://sea2.discourse-cdn.com/grafana/user_avatar/community.grafana.com/jangaraj/32/39197_2.png) [@jangaraj](https://community.grafana.com/u/jangaraj)\ **Post date:** [July 17, 2024, 10:55am UTC](https://community.grafana.com/t/request-canceled-client-timeout-exceeded-while-awaiting-headers/126348/10 "2024-07-17T10:55:27Z")
I don’t understand your query/metric logic, so I don’t know. ---
**Author:** ![saravahdatipour](https://avatars.discourse-cdn.com/v4/letter/s/bc79bd/32.png) [@saravahdatipour](https://community.grafana.com/u/saravahdatipour)\ **Post date:** [April 2, 2025, 9:13am UTC](https://community.grafana.com/t/request-canceled-client-timeout-exceeded-while-awaiting-headers/126348/11 "2025-04-02T09:13:15Z")
@delaramhamraz Hi, did you end up figuring out what the issue was? We’re having the exact same problem, and our Loki and Grafana is externally managed by our cloud provider and the support team can’t find anything wrong with it, yet we’re experiencing these timeouts. Sometimes they don’t timeout, but it’s just very inconsistent and unreliable. ---
**Author:** ![davidallen5](https://sea2.discourse-cdn.com/grafana/user_avatar/community.grafana.com/davidallen5/32/32092_2.png) [@davidallen5](https://community.grafana.com/u/davidallen5)\ **Post date:** [June 20, 2025, 1:51pm UTC](https://community.grafana.com/t/request-canceled-client-timeout-exceeded-while-awaiting-headers/126348/12 "2025-06-20T13:51:16Z")
This topic has been open for a while so I wanted to try to summarize the best available advice; from what others have said above and some extra things. This is intended to help the next person who finds this thread. Recommendations if you’re in this situation are the following, but what will work for you will depend on your situation and specific query. - **Optimize Query:** Simplify the Prometheus query to reduce processing time and resource usage. OP in this thread didn’t have an option to do this, but if you do, you should. ([Here’s information on how to optimize](https://valyala.medium.com/how-to-optimize-promql-and-metricsql-queries-85a1b75bf986) - **Adjust Grafana Settings:** Ensure Grafana’s timeout settings are appropriately configured in both `grafana.ini` and the data source settings. - **Use Recording Rules:** Consider using Prometheus recording rules to precompute and store results, reducing query load. [Recording rules docs](https://grafana.com/docs/grafana/latest/alerting/alerting-rules/create-recording-rules/). - **Make sure CPU & RAM is sufficient** Check the stats on your servers that they’re healthy - **Adjust query interval:** Balancing granularity and performance, basically - **Explore Query Sharding:** Investigate if query sharding or breaking down the query into smaller parts can help. [See also this](https://medium.com/@Nitish_Mane/scaling-prometheus-with-thanos-a-guide-to-long-term-scalable-monitoring-a1eca334cbd4). Basically all of these options attack all of the potential causes and degrees of freedom you have to solve the problem.