Hi, everyone. I had a problem with grafana and nginx which really need your help. I have two grafana servers on different node and use nginx as lb, and I also have a TV monitor showing some dashboards, the TV monitor running all the time with refresh interval set to 15s. Last day I found the dashboard unable to work, I used the following command to checked the port on grafana server:
netstat -nutp | grep 3000 | grep ESTABLISHED | wc -l
and found almost 10k connections on port 3000(which is grfana server port). Following command shows that there are too many open files:
systemctl status grafana-server.service
I also check the nginx logs which show all the connections come from the TV monitor.
After restart the grafana service connections were reset to 0 but I found it increase slowly and never drop down, after 2 days number of connections increased to 6k+. It seemes that like connections between nginx and grafana server never released event the requests are done.
I found some cases in google which are similiar to mine, but I don’t think they can help.
https://github.com/grafana/grafana/issues/8513, this one says the max_idle_conn need to be set to something greater than 0, and I found this parameter is set to 2 in my config file.
https://github.com/grafana/grafana/issues/6759, this one showes the 4.0.0 version has this problem and suggest to upgrade grafana’s version, but my grafana is version 5.4.0.
I’m not sure this is an nginx problem or a grafana problem. Am I config something wrong in grafana or nginx? Really need your help. Thank you!