Grafana, loki and promtail running, connection to loki refused

Grafana Loki
1

Hi there.
I am testing a way to monitor Mikrotik routers through Grafana/Loki

Running on:
Lenovo h515s
AMD A6-5200
8GB DDR3
Wired network directly to testing Mikrotik - all security except outside access and wifi is relaxed

Rocky Linux 9.4
kernel 5.14.0-427.16.1.el9_4.x86_64
Mate 1.26 (going to run gui-less but gui for easier command input)

Running through Docker CE

My Grafana docker log:

logger=tsdb.loki endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=checkHealth pluginId=loki dsName=loki dsUID=ae3jxonhhrg8wd uname=admin fromAlert=false t=2024-11-10T20:09:57.926196168Z level=info msg="Prepared request to Loki" duration=77.432µs queriesLength=1 stage=prepareRequest runInParallel=false
logger=tsdb.loki endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=checkHealth pluginId=loki dsName=loki dsUID=ae3jxonhhrg8wd uname=admin fromAlert=false t=2024-11-10T20:09:57.927845823Z level=error msg="Error received from Loki" error="Get \"http://localhost:3100/loki/api/v1/query?direction=backward&query=vector%281%29%2Bvector%281%29&time=4000000000\": dial tcp [::1]:3100: connect: connection refused" status=error duration=1.474388ms stage=databaseRequest start=1970-01-01T00:00:01Z end=1970-01-01T00:00:04Z step=1s query=vector(1)+vector(1) queryType=instant direction=backward maxLines=0 supportingQueryType=none lokiHost=localhost:3100 lokiPath=/loki/api/v1/query
logger=tsdb.loki endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth t=2024-11-10T20:09:57.928158926Z level=error msg="Loki health check failed" error="error from loki: Get \"http://localhost:3100/loki/api/v1/query?direction=backward&query=vector%281%29%2Bvector%281%29&time=4000000000\": dial tcp [::1]:3100: connect: connection refused"
logger=context userId=1 orgId=1 uname=admin t=2024-11-10T20:09:57.928318683Z level=info msg="Request Completed" method=GET path=/api/datasources/uid/ae3jxonhhrg8wd/health status=400 remote_addr=172.17.0.1 time_ms=9 duration=9.435105ms size=106 referer=http://localhost:3000/connections/datasources/edit/ae3jxonhhrg8wd handler=/api/datasources/uid/:uid/health status_source=server

my Loki config:

---
server:
  http_listen_address: 0.0.0.0
  http_listen_port: 3100

memberlist:
  join_members: ["read", "write", "backend"]
  dead_node_reclaim_time: 30s
  gossip_to_dead_nodes_time: 15s
  left_ingesters_timeout: 30s
  bind_addr: ['0.0.0.0']
  bind_port: 7946
  gossip_interval: 2s

schema_config:
  configs:
    - from: 2023-01-01
      store: tsdb
      object_store: s3
      schema: v13
      index:
	prefix: index_
        period: 24h
common:
  path_prefix: /loki
  replication_factor: 1
  compactor_address: http://backend:3100
  storage:
    s3:
      endpoint: minio:9000
      insecure: true
      bucketnames: loki-data
      access_key_id: loki
      secret_access_key: supersecret
      s3forcepathstyle: true
  ring:
    kvstore:
      store: memberlist
ruler:
  storage:
    s3:
      bucketnames: loki-ruler

(lightly modified default config)

The default promtail config:

                                                                         
server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
- job_name: system
  static_configs:
  - targets:
      - localhost
    labels:
      job: varlogs
      __path__: /var/log/*log

and a modified one I found online for the things with Mikrotik logs sent through rsyslog(not written completely):

server:
  http_listen_port: 9080
  grpc_listen_port: 0
        
positions:
  filename: /tmp/positions.yaml
        
clients:
  - url: http://loki:3100/loki/api/v1/push
        
scrape_configs:
- job_name: system
  static_configs:
  - targets:
      - localhost
    labels:
      job: mikrotik
      __path__: /var/log/*

This exact error(connection refused) happens no matter if I keep the configs default, fix the errors the logs tell me or use the custom configs.

I tried using the old version from the starter guide on the Grafana website. Copying steps exactly. That gives xychart errors or failed to find open plugin path errors.

All containers are running, Loki loads the ready page and metric page. Grafana of course works too.

CONTAINER ID   IMAGE                     COMMAND                  CREATED       STATUS          PORTS                                       NAMES
18c2af798862   grafana/grafana-oss       "/run.sh"                3 hours ago   Up 3 hours      0.0.0.0:3000->3000/tcp, :::3000->3000/tcp   grafana
ed4c153d7dd8   grafana/promtail:latest   "/usr/bin/promtail -…"   3 hours ago   Up 23 minutes                                               promtail
3ea3c02ee31e   grafana/loki:latest       "/usr/bin/loki -conf…"   3 hours ago   Up 23 minutes   0.0.0.0:3100->3100/tcp, :::3100->3100/tcp   loki

all configs are in a specified folder. Configs are indeed working.

Is there a specific setup step I missed?

Any help for me, as a network beginner is appreciated. I am still grasping all the network stuff after only working on computers themselves all the time. I myself did not really expect to start doing network stuff too.

2
  1. Where are you getting connection refused error? From promtail to Loki, or Grafana to Loki?
  2. Test from the origin of error (let’s say promtail), for things like DNS resolution and telnet.
  3. I suspect your containers may not be on the same docker network (meaning http://loki is probably not resolving). And if this is the case you also cannot establish connections between containers directly.