Unexpected EOF when docker host is behind a socket proxy

Grafana Alloy
, ,
1

Hello all. I’m seeing the following warning logged repeatedly:

{"ts":"2025-01-17T21:02:14.323065311Z","level":"warn","msg":"could not transfer logs","component_path":"/","component_id":"loki.source.docker.default","target":"docker/54ad36a05cb107746b55522d8e7c875484e181f47d37044694db7c20375557c6","written":0,"container":"54ad36a05cb107746b55522d8e7c875484e181f47d37044694db7c20375557c6","err":"unexpected EOF"}

I still see logs flowing through to Loki and Grafana, but the warning shows up pretty regularly, referring to several containers, not just the single one in the above example. Here’s an illustration of the occurrence in a one hour range:

image
image701×210 8.44 KB

I’ve narrowed this down to only occurring when using a “socket proxy” for the docker host. So instead of bind mounting /var/run/docker.sock on the host node to the container, I refer to it with “tcp://socket-proxy:2375” which is a separate container that’s basically running HAProxy with a bunch of rules. This problem goes away when I switch to bind mounting /var/run/docker.sock and referring to “unix:///var/run/docker.sock” in the configuration instead.

Now, when I look up the associated log (based on the timestamp) in the proxy container, I see that the request isn’t filtered out (i.e. status 200), but there are termination flags present that seems to explain why Alloy is logging the warning.

::ffff:192.168.255.4:51338 [17/Jan/2025:20:52:14.318] dockerfrontend dockerbackend/dockersocket 0/0/0/1/600004 200 1638 - - sD-- 9/9/6/6/0 0/0 "GET /v1.46/containers/54ad36a05cb107746b55522d8e7c875484e181f47d37044694db7c20375557c6/logs?follow=1&since=1737044497&stderr=1&stdout=1&tail=&timestamps=1 HTTP/1.1"

Note the sD--. According to the reference that I linked above, the reason seems to be that:

  • “the server-side timeout expired while waiting for the server to send or receive data.”
  • “the session was in the DATA phase.”

I’m honestly not familiar enough with the HAProxy termination flags to understand what’s really going on here. I don’t know if this is due to a missing configuration on Alloy and/or the socket proxy. Or perhaps something else entirely in my environment. FWIW, I also have cAdvisor and Traefik containers also referring to the same socket proxy and I have yet to see any warnings or issues there.

I’m also aware this could really just be a problem with the socket proxy project, but figured I’d ask here first. I’ll open an issue over there if that ends up being the advice here.

For reference, here are some of my relevant configs. Please feel free to ask for other configs that might be helpful.

socket proxy container service section 
---
services:
  socket-proxy:
    image: docker.io/tecnativa/docker-socket-proxy:0.3

    environment:
      - LOG_LEVEL=info

      # Granted by default
      - EVENTS=1
      - PING=1
      - VERSION=1

      # Critical
      - AUTH=0
      - SECRETS=0
      - POST=0

      # Use caution
      - CONTAINERS=1
      - IMAGES=1
      - INFO=1
      - NETWORKS=1

    restart: unless-stopped

    security_opt:
      - no-new-privileges=true

    volumes:
      - type: bind
        source: /var/run/docker.sock
        target: /var/run/docker.sock
        read_only: true
config.alloy 
logging {
  level  = "info"
  format = "json"
}

discovery.docker "default" {
  host             = "tcp://socket-proxy:2375"
  refresh_interval = "1m"
}

discovery.relabel "docker" {
  targets = []

  rule {
    target_label = "job"
    replacement  = "integrations/docker"
  }

  rule {
    target_label = "instance"
    replacement  = constants.hostname
  }

  rule {
    source_labels = ["__meta_docker_container_log_stream"]
    target_label  = "stream"
  }

  rule {
    source_labels = ["__meta_docker_container_name"]
    regex         = "/(.*)"
    target_label  = "container"
  }
}

loki.source.docker "default" {
  targets          = discovery.docker.default.targets
  relabel_rules    = discovery.relabel.docker.rules
  host             = "tcp://socket-proxy:2375"
  refresh_interval = "1m"

  labels = {
    "service_name" = "docker",
  }

  forward_to = [
    loki.write.default.receiver,
  ]
}

loki.write "default" {
  endpoint {
    url = "http://loki:3100/loki/api/v1/push"
  }
}
2

I could only include two links in my post due to having just created my account here today. I also meant to include the link to the HAProxy config that’s used in the socket-proxy. The configuration is purely done through environment variables (see service definition in my original post).

4

Looks like this is very likely normal behavior actually. Found a closed issue from a few years ago in the socket-proxy repository that looks pretty accurate to what I’m experiencing. The “problem” is simply lack of active logs from several of my containers. So not really a problem.

I guess I can try increasing the timeout on the HAProxy side, but that just prolongs the warnings from showing up as often. Since this is out of scope of Alloy anyway, I’ll mark this as solved.