We have a company-wide Grafana Cloud instance with its Prometheus and AlertManager, that many teams use (the policy is that each team can use whatever tools they want). We also have a self-hosted Prometheus/AlertManager instance that we’re managing via Infrastructure as Code (IaC).
One of the teams that are using the Cloud Prometheus has passed on the management of their infrastructure to my team. We don’t want to migrate them to using our self-hosted Prometheus, since they have already set up a bunch of dashboards that reference the Cloud Prometheus. However, we would like to manage alerts with our IaC deployment of AlertManager.
We can’t scrape the application directly from our Prometheus instance, since it’s an auto-scaled K8s cluster that doesn’t have static IP addresses.
What we would ideally like, is to hook up the Cloud Prometheus instance to our self-hosted AlertManager, but I haven’t seen a way to do it.
I was thinking that we might use Prometheus Federation to pull the metrics from the cloud instance and push them to our self-hosted Prometheus instance. Would that work? What would be the federation endpoint for Cloud Prometheus?
Otherwise we would probably have to create a sidecar that scrapes the metrics from the application, and pushes them to our Push Gateway.
Any other ideas?