I am attempting to set up alerting for when my Grafana server goes down on box1 from box2 (in same LAN).
I tried simply using the Prometheus metrics exposed on the :3000/metrics url. I note that Prometheus itself has no problem scraping this url, from either box1 or box2.
However, trying all the various permutations of this url (with and without the http, with and without the /metrics directory appended to the url) I cannot add the data source to Grafana on box2.
In particular, I am able to successfully add box1’s Prometheus (:9090) in the box2 Grafana. So its not a firewall or DNS issue. But for some reason box2’s Grafana won’t add box1’s Grafana/metrics (the prometheus metrics exposed by box1’s Grafana) as a data source and the error message is rather limiting: HTTP Error Not Found.
Is there something I’m missing? How do most people receive an alert when their main Grafana goes down?