Grafana Alloy prometheus.scrape federation with another Prometheus instance

Grafana Alloy
1

Hi,

BACKGROUND
I’m trying to get the Grafana Alloy agent working with collecting exporter metrics provided by Docker Swarm containers, and was unsuccessful when trying to use the discovery.dockerswarm > prometheus.scrape component workflow.

I think this was because the Alloy agent is running on each Docker host as a systemd service, and it doesn’t have connectivity to the Docker services’ networks (they expose app metrics on an unpublished / overlay ‘monitoring’ network). Historically, we scrape these metrics using a Prometheus swarm container, that is connected to the ‘monitoring’ swarm network, then in turn scrape its’ metrics via Prometheus federation to another standalone Prometheus server. I wasn’t able to find a solution to get Alloy to work here.

CURRENT ISSUE
I’ve pivoted to trying to get the Alloy agent to scrape the swarm-based Prometheus ‘/federate’ metrics path, but I am struggling with the URL encoding.

I have tried using both the
metrics_path = /federate?match[]=%7B__name__%3D~%22.%2B%22%7D
and also the
params = { "match[]" = ["%7B__name__%3D~%22.%2B%22%7D"] }
arguments within the prometheus.scrape block, however both methods substitute the [ ] in the 'match[ ]' with %5B therefore the scrape doesn’t want to work i.e. last_error "server returned HTTP status 404 Not Found"

Please help!

2

Apologies, ended up getting it working by using the alloy convert command, and converting a snippet of Prometheus federation scrape config into Alloy config.

Working code is as follows:

prometheus.scrape "dockerswarm_prometheus_services" {
  job_name   = "dockerswarm_discovery"
  metrics_path = "/federate"
  honor_labels = true
  honor_timestamps = true
  scrape_interval = "30s"
  scrape_timeout = "20s"
  scheme = "http"
  follow_redirects = true
  params       = {
                  "match[]" = ["{job=~\".*\"}", "{__name__=~\"^job:.*\"}"],
          }
  targets    = discovery.relabel.dockerswarm_prometheus_services.output
  forward_to = [prometheus.remote_write.prometheus.receiver]
}