How to collect logs of services (application) from all Docker Swarm nodes with Alloy?

Grafana Alloy
1

I’m running a Docker Swarm cluster and want to use Grafana Alloy to send container logs to Loki.

if we use docker discovery I think we will have an issue (The container name changes on every restart or reschedule)

  1. Do I need to run Alloy on every Swarm node (as a global service) to collect logs cluster-wide?
  2. Is there a way to configure Alloy so that it can pull logs for all Swarm services directly from the manager node, like docker service logs does?

Thanks!

3

I don’t use Docker Swam myself, but I would recommend you to run alloy on all nodes.

4

Can anyone using alloy to monitor a docker swarm cluster help me by sharing the config?
I’m using:

discovery.dockerswarm "swarm" {
  host = "unix:///var/run/docker.sock"
  role = "tasks"

  filter {
    name   = "desired-state"
    values = ["running"]
  }
}

and targets are discovered correctly.

After this, I tried with component loki.source.docker:

loki.source.docker "swarm_reader" {
  host = "unix:///var/run/docker.sock"
  targets = discovery.dockerswarm.swarm.targets
  forward_to = [loki.write.endpoint.receiver]
}

but no logs send to loki.

So I tried with loki.source.file with custom __path__ from swarm metadata:

discovery.relabel "swarm_logs" {
  targets = discovery.dockerswarm.swarm.targets
  rule {
    source_labels = ["__meta_dockerswarm_task_container_id"]
    target_label  = "__path__"
    replacement   = "/var/log/containers/${1}-json.log"
  }

  rule {
    source_labels = ["__meta_dockerswarm_service_name"]
    target_label  = "service"
  }
}

loki.source.file "swarm_tasks" {
  targets    = discovery.relabel.swarm_logs.output
  forward_to = [loki.write.endpoint.receiver]
}

but no logs send to loki.

Anyone could help me?

5

Please start a new thread. Also, more information would be great, such as if there were error logs from Alloy and if any other pipeline is working.