Hello, I am exploring the usage of otel prometheus receiver vs prometheus scrape component in my setup.
Current setup:
- Microservices are instrumented with Prometheus client library and are exposing metrics in Prometheus format
- There is a central Prometheus backend
- The central Prometheus server is directly scraping metrics from all hosts - a small test setup of 2 host with 10x microservices.
Requirement: Scale the setup for prod - 10s of microservices on 100s of hosts.
Possible scraping setup:
It seems efficient to run a light-weight Prometheus agent on each host machine that just scrapes and sends the data to remote Prometheus server - could be distributed and horizontally scaled using Thanos/mimir in future
Understanding:
- Prometheus agent mode is interesting but still experimental
- Prometheus federation setup would be heavyweight and overkill
- Grafana alloy prometheus scrape or prometheus otel collector component can cater to the usecase
Query:
- Can you please suggest which would be the right component for such a usecase ?
- Does otel collector convert the prometheus metrics to opentelemetry metrics implicitly. If yes, what is the performance impact ?