Hello,
I have a problem with my alerts, they are all finishing with timeout when it tries to reach graphite server:
firing:true
state:“alerting”
conditionEvals:" = true"
timeMs:“1.116ms”
error:“tsdb.HandleRequest() error Post http://10.118.248.94/render: net/http: request canceled (Client.Timeout exceeded while awaiting headers)”
I can confirm that graphite is reachable, but without proxy, as it is defined in the datasource:
I wonder if grafana is using the system proxy configuration in the alert process because I have a proxy configured and graphite is not reachable through it.
Can you confirm if this is a normal behavior?
Thank you in advance.
Hi,
I think you need to use proxy mode in your datasource for alerting to work.
Marcus
Direct mode in this case means direct from your browser to the data source. Proxy means going via the grafana backend to the data source. Proxy mode is the default and direct mode is only needed in a few special cases like single sign authentication via the browser. By proxying the request through Grafana, it can add stuff like CORS headers which you would have to add yourself via a proxy like nginx otherwise.
Hello, thanks for your answers. Actually, my only way to reach graphite is using direct mode, so I think alert system should use the same configuration.
As you are going directly from your browser to the data source, then alerting won’t work. If the alerting functionality was written in the Grafana frontend, then it would stop working if you closed your browser. It has to run on the backend and therefore only works for proxy mode connections.
I implemented a workaround configuring exceptions in proxy configuration.
Thanks for your help!
