Grafana too many open files error

Hi, everyone. I had a problem with grafana and nginx which really need your help. I have two grafana servers on different node and use nginx as lb, and I also have a TV monitor showing some dashboards, the TV monitor running all the time with refresh interval set to 15s. Last day I found the dashboard unable to work, I used the following command to checked the port on grafana server:
netstat -nutp | grep 3000 | grep ESTABLISHED | wc -l
and found almost 10k connections on port 3000(which is grfana server port). Following command shows that there are too many open files:
systemctl status grafana-server.service
I also check the nginx logs which show all the connections come from the TV monitor.
After restart the grafana service connections were reset to 0 but I found it increase slowly and never drop down, after 2 days number of connections increased to 6k+. It seemes that like connections between nginx and grafana server never released event the requests are done.

I found some cases in google which are similiar to mine, but I don’t think they can help., this one says the max_idle_conn need to be set to something greater than 0, and I found this parameter is set to 2 in my config file., this one showes the 4.0.0 version has this problem and suggest to upgrade grafana’s version, but my grafana is version 5.4.0.

I’m not sure this is an nginx problem or a grafana problem. Am I config something wrong in grafana or nginx? Really need your help. Thank you!

check out the above solutions.