Promtail error sending batch (to loki), connection refused

Grafana Loki
1

There are a few other posts with similar issues but none of those solutions are working for me.

I’m running the following docker-compose.yaml here to launch grafana, promtail and loki.

version: "3"

networks:
  loki:

services:
  loki:
    image: grafana/loki:2.9.4
    ports:
      - "3100:3100"
    command: -config.file=/etc/loki/local-config.yaml
    networks:
      - loki

  promtail:
    image: grafana/promtail:2.9.4
    volumes:
      - /var/log:/var/log
      - ~/.ros/log/latest:/ros/log
      - ~/grafana/promtail:/promtail
    command: -config.file=/promtail/promtail.yaml
    networks:
      - loki

  grafana:
    environment:
      - GF_PATHS_PROVISIONING=/etc/grafana/provisioning
      - GF_AUTH_ANONYMOUS_ENABLED=true
      - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
    entrypoint:
      - sh
      - -euc
      - |
        mkdir -p /etc/grafana/provisioning/datasources
        cat <<EOF > /etc/grafana/provisioning/datasources/ds.yaml
        apiVersion: 1
        datasources:
        - name: Loki
          type: loki
          access: proxy
          orgId: 1
          url: http://loki:3100
          basicAuth: false
          isDefault: true
          version: 1
          editable: false
        EOF
        /run.sh
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    networks:
      - loki

my promtail config is

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://127.0.0.1:3100/loki/api/v1/push

scrape_configs:
 - job_name: system
   pipeline_stages:
   - docker:
   static_configs:
   - targets:
      - localhost
     labels:
      job: varlogs
      host: yourhost
      __path__: /var/log/*.log

 - job_name: ros_job
   static_configs:
   - targets:
      - localhost
     labels:
      job: roslogs
      host: yourhost
      __path__: /ros/log/*.log

Promtail is picking up all my log files ok but in the docker-compose output its failing to push the logs to loki

promtail-1  | level=warn ts=2024-02-29T06:22:39.263384646Z caller=client.go:419 component=client host=127.0.0.1:3100 msg="error sending batch, will retry" status=-1 tenant=1 error="Post \"http://127.0.0.1:3100/loki/api/v1/push\": dial tcp 127.0.0.1:3100: connect: connection refused"

I’m able to curl the loki/ready address from my terminal

curl 127.0.0.1:3100/ready
ready

but if i try and curl from the promtail container I get the connection refused. Seems like its a docker networking issue?

2

If you are creating a network in docker compose, shouldn’t you be using loki:3100 to connect from promtail to Loki?

Remember, 127.0.0.1 on your host is different from 127.0.0.1 on your container.

1 Like
3

Ahh yes that fixed it. Thank you!