We have a number of different services in kubernetes that we would like to setup monitoring for. I’ve found that a lot of helm charts have options to install prometheus and grafana that are pre-configured for that service. This makes the setup of monitoring for that service easier. This approach also means that you end up with multiple copies of prometheus and grafana running.
My question is, within a kubernetes cluster what is the best practice for installing prometheus and grafana?
Should there be a single installation of prometheus and grafana that all other services register with, or
Should you allow each service to install their own instance of prometheus and grafana.
My initial thoughts would be to use a single installation that all other services would register their metrics endpoints with but I’m not sure if this is correct? Are there limitations with using a single installation that I need to be aware of?