Hello All, I got the Error: Too many connections in Grafana alert rules.
And Error log in grafana:
logger=ngalert.scheduler rule_uid=HEsLxEU4z org_id=1 version=3 attempt=0 now=2023-05-24T00:00:10Z t=2023-05-24T00:00:11.452364707Z level=error msg="Failed to build rule evaluator" error="failed to build query 'A': Error 1040: Too many connections" logger=ngalert.state.manager rule_uid=HEsLxEU4z org_id=1 t=2023-05-24T00:00:11.46356797Z level=error msg="Failed to save alert states" states="[{State:Error Labels:__alert_rule_namespace_uid__=W8GWe4V4z, __alert_rule_uid__=HEsLxEU4z, alertname=Task Max Retry FRA, grafana_folder=WxC-UDP}]" error="Error 1040: Too many connections" logger=ngalert.state.historian backend=annotations rule_uid=HEsLxEU4z org_id=1 t=2023-05-24T00:00:11.475806135Z level=error msg="Error getting dashboard for alert annotation" dashboardUID=v2IhrcJVk error="Error 1040: Too many connections" logger=ngalert.state.historian backend=annotations rule_uid=HEsLxEU4z org_id=1 t=2023-05-24T00:00:11.486972393Z level=error msg="Error saving alert annotation batch" error="Error 1040: Too many connections" logger=datasources t=2023-05-24T00:00:12.881035754Z level=error msg="Failed getting data source" err="Error 1040: Too many connections" uid=9jPXcGcnz id=0 name= orgId=1 logger=ngalert.scheduler rule_uid=_ujGemsVk org_id=1 version=41 attempt=0 now=2023-05-24T00:00:10Z t=2023-05-24T00:00:12.881159219Z level=error msg="Failed to build rule evaluator" error="failed to build query 'A': Error 1040: Too many connections" ....
What Grafana version are you using?
Grafana 9.4.7
What datasource are you using?
Grafana stores：mysql:8.0.18
What OS are you running grafana on?
Docker
Part of the grafana.ini file, sensitive information is indicated by :
[server]
;protocol = http
;http_addr =
;http_port = 3000
domain = grafana.***.com
;enforce_domain = false
root_url = %(protocol)s://%(domain)s/
;router_logging = false
;static_root_path = public
;enable_gzip = false
;cert_file =
;cert_key =
[database]
type = mysql
host = ***********************
name = grafana-prod
user = grafana
password = *************
# Max idle conn setting default is 2
max_idle_conn = 2
# Max conn setting default is 0 (mean not set)
max_open_conn =
# Connection Max Lifetime default is 14400 (means 14400 seconds or 4 hours)
conn_max_lifetime = 14400
Grafana stores：MySQL max_open_conn is set to 151 and wait_timeout is 28800
Why is the error 1040: Too many connections, is the connection limit of max_idle_conn in grafana.ini or other places.
How to solve it? Thank you