Nodes not remove from Grafana Dashboard while remove from prometheus config file

  • What Grafana version and what operating system are you using?
    10.3.3 and amazon linux 2

  • What are you trying to achieve?
    configure the dashboard using prometheus and node exporter

  • How are you trying to achieve it?
    create a standalone server

  • What happened?
    nodes already remove from prometheus config file but still showing in grafana dashboard.

  • What did you expect to happen?

  • Can you copy/paste the configuration(s) that you are having problems with?

  • Did you receive any errors in the Grafana UI or in related logs? If so, please tell us exactly what they were.

  • Did you follow any online instructions? If so, what is the URL?

Firstly, after removing the nodes from the config file are you restarting the Prometheus service/s?

But even when you do this, if there is old data from the nodes still in Prometheus then Grafana will continue to show the data. Depending on how many days of data you have configured Prometheus to keep then the data will disappear from Grafana after that number of days.

I don’t actually know how to manually remove specific node data from Prometheus, in my case I just flush the entire Prometheus data as it’s just for my use and I can clear history from time to time without a problem. If you can do the same then my hacky way to clear ALL Prometheus metrics is this:

rm -rf /var/lib/prometheus/metrics2/*