Grafana Agent exposing collected metrics over http

Grafana Grafana Agent
1

Hi,

I’m trying to setup the Grafana Agent (on-premise) in the following way:

  • Processes are sending metrics with OTLP to Grafana Agent.
  • Grafana Agent collects them and exposes them to Prometheus.
  • Prometheus scrapes the data from the Grafana Agent (from a defined port/path).

I’m exploring the Grafana Agent in flow mode. Unfortunately, all the components related to OTel and Prometheus end up sending the data synchronously with remote write API.

I’m missing something here, or was the Grafana Agent not designed for such flows?

Best,
Marek Mierzwa

2

I would say that’s approach from Prometheus world. Newer OTEL approach is pushing instead of pulling data. It scales better.

I would use use OTEL approach:

  • Processes are sending metrics with OTLP to local OTEL collector (service)
  • OTEL collector (service) pushes them to Prometheus
1 Like
3

Hi Jangoraj! Thank you for your quick reply :blush:

Can you elaborate more on the scalability improvement of pushing telemetry vs. pulling from collector/agent? Or point me to some source/discussion?

I’d also love to learn if this is just a preferable way or if exposing the metrics data is just not designed into the Grafana Agent?

Best,
Marek

4

See Using OpenTelemetry and Prometheus: A practical guide to data collection
Point 6

1 Like
5

When the collector/Agent (or a cluster of them) makes the most of the work, it makes sense indeed. Thank you for the explanation :blush:

Best,
Marek