I also notice that when grafana goes to 3.21 M/s or more this message starts to appear in the log file level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=1 code="database is locked"
i removed all prometheus targets and leaved only one job from node_exporter to see is something is changed, i hoped when prometheus have low amount of data to collect and send the write speeds of grafana will go down because there is less information to write.
I guess you are using the default SQLite DB (which is a terrible choice for high traffic, but people are using it because “it just works” by default). Serious Grafana deployments use more serious DB (MySQL, PotgreSQL)