Getting : Status: 504. Message: Post "http://localhost:9090/api/v1/query_range": net/http: timeout awaiting response headers for cpu metrics

Grafana Dashboards
,
1

Getting : Status: 504. Message: Post “http://localhost:9090/api/v1/query_range”: net/http: timeout awaiting response headers
error in Grafana dashboard while querying cpu utilisation of my servers.

this error comes only when the time period of data is set to 30days or more none of the other other metrics such as network , memory are showing this error.

Screenshot from 2023-08-18 12-00-57
Screenshot from 2023-08-18 12-00-571292×551 70.8 KB

similarly getting

504 Gateway Time-out

504 Gateway Time-out

nginx/1.18.0 (Ubuntu)

Object

status:504

statusText:“Gateway Time-out”

data:Object

message:" 504 Gateway Time-out

504 Gateway Time-out

nginx/1.18.0 (Ubuntu) "

error:“Gateway Time-out”

response:" 504 Gateway Time-out

504 Gateway Time-out

nginx/1.18.0 (Ubuntu) "

config:Object

url:“api/ds/query?ds_type=prometheus&requestId=Q207”

method:“POST”

data:Object

requestId:“Q207”

hideFromInspector:false

headers:Object

retry:0

traceId:undefined

message:" 504 Gateway Time-out

504 Gateway Time-out

nginx/1.18.0 (Ubuntu) "

2

Hello,

This could be related to NGINX configs.

Check this

In your /etc/nginx/sites-available/default or /etc/nginx/nginx.conf add the following variables:

proxy_connect_timeout       240;
proxy_send_timeout          240;
proxy_read_timeout          240;
send_timeout                240;

Run sudo nginx -t to check the syntax, and then sudo service nginx restart.

This should effectively quadruple the time before NGINX will timeout your API requests (the default being 60 seconds, our new timeout being 240 seconds).

3

Thanks @clevernessisamyth

This error wen away when i increased the time out period of my data source , since I’m monitoring huge application it was taking heavy load on it

1 Like