what is the recommended setup to do HA Prometheus + Grafana? With Grafana itself it sounds easy: sticky sessions, shared HA database. But what to do with the Prometheus datasource?
Promemetheus itself supports only HA by duplicating the installation.
Should I setup for each grafana a different datasource? - for Grafana A setup Prom A, for Grafana B setup Prom B. Would it even be possible with the shared SQL database?
Or should I put Prometheus API behind load balancer? - will this work, when the prometheus A and B will have different almost always slightly different data (because scrapes on A and B are not synchronized in any way)? The only possibility which occurs to me is not to use round robin on the Prometheus balacer, but use one of the servers as a Hot backup…
Is there any other way?