I am using grafana to monitor my Jenkins instances deployed in kubernetes. The metrics that I get are like:
jenkins_health_check_score{instance="172.24.11.212:8090",job="jenkins-metrics",namespace="test"}
So every Jenkins Master pod gets an instance deployed in a namespace.
Prometheus is my data source and I can see the metrics in my Jenkins Dashboard correctly.
The issue occurs when I deploy a lot of Jenkinses and delete them. Grafana seems to remember all the instances even after they are deleted. Meaning that if Im using a filter on instance, the drop down menu will contains IPs of ALL the jenkins I created and then deleted. Meaning the drop down list gets longer and longer and longer.
And shows outdated data. Is this data being cached ? Is there a way to delete this cache or this data so that I only see instances that are currently live and not the ones that were deleted ages ago ?