Prometheus HA setup

Grafana Prometheus
,
1

Hi,

Prometheus container image+version: prom/prometheus:v3.0.0

We’re running an AWS EKS cluster with 3 worker nodes with an APM set up (OTEL, Grafana, Jaeger & Cloudwatch log).

We’ve managed to successfully make all services highly availably except for Prometheus where its only running a single pod/container as we had routinely received issues about multiple pods unable to take a hold of the time series database that Prometheus has and the pods would shut down.

In order to combat the problem, we configured Thanos where metrics older than 2 hours in Prometheus are pushed into an AWS S3 bucket automagically and said metrics are retrieved through the Thanos query pods (not issues here, it works perfectly).

If Prometheus has issues and stops or if the node it’s running on falls over, there’ll be a brief moment in time where Prom is down as it’s currently only on a single pod/container.

Is anyone able to advise if there’s a certain AWS Service or backend storage that allows Prometheus to run on multiple pods without issues? We currently host the Prometheus TSBDB on AWS EFS (I know it’s not recommended)

Cheers!

2

Hi,
Have you maybe checked VictoriaMetrics (especially the cluster version). It’s not prometheus but it would allow you to run multiple agents (scraping pods) and remote_write metrics directly to Thanos (or VictoriaMetrics storage if you decide to go all the way in). That way there wouldn’t be the any automagic in here:

Thanos where metrics older than 2 hours in Prometheus are pushed into an AWS S3

Since VM would automatically push data to Thanos. It’s not Prometheus, so some migration might be in order, but VM is prometheus-compatible.