I was wondering if there is anyone have encountered this issue.
I am implementing a health check metrics through Prometheus + Push Gateway + Grafana
I am unable to use the scrape from Prometheus because the metrics are unreachable from outside so we implemented a Push Gateway for it.
Now I’m using a Singlestat Panel to show Healthy/Unhealthy metric
Now the issue arises when the servers that send the metrics are actually down or offline.
The metrics at Prometheus will still be there and Grafana will in turn show it as Healthy, which it’s not.
And as far as I can search, Prometheus decided to not implement any Push Gateway metric timeout feature. Here.
Is there any way for Grafana to filter out any stale metrics ?
Or any workaround for my use case ?