Loki and Promtail working but can't connect from Grafana

[meddars]

I’m struggling to get this working for the first time using docker.

promtail config.yml

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /positions/positions.yaml

clients:
  - url: http://localhost:3100/loki/api/v1/push

scrape_configs:
- job_name: system
  static_configs:
  - targets:
      - localhost
    labels:
      job: varlogs
      __path__: /var/log/*log

Loki local-config.yaml

auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9096

common:
  path_prefix: /tmp/loki
  storage:
    filesystem:
      chunks_directory: /tmp/loki/chunks
      rules_directory: /tmp/loki/rules
  replication_factor: 1
  ring:
    instance_addr: 127.0.0.1
    kvstore:
      store: inmemory

schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h

ruler:
  alertmanager_url: http://localhost:9093

When I try and connect from Grafana to the Loki data source I get this error:

I see this in the Grafana log:

logger=context userId=1 orgId=1 uname=admin t=2022-11-01T21:20:38.419418035Z level=error msg="Failed to call resource" error="Get \"http://localhost:3100/loki/api/v1/labels?start=1667337038277000000&end=1667337638277000000\": dial tcp [::1]:3100: connect: connection refused" traceID=

Why is it refused? My containers are all on the same box.

[athakur]

Can you share either the Docker compose file or the full docker command line you’re using to start Loki and Promtail? Also is Grafana also running as a container?

My initial thought is that unless all of the containers are running with “–net=host” (or the equivalent Docker Compose configuration), you aren’t going to be able to use “localhost” to communicate between containers. Each container normally runs in it’s own network space and as such, “localhost” points back to that specific container.

[meddars]

The containers are running on Unraid so not sure what the docker run commands are. Does this help? BTW I changed the IP address in the config file to the host IP based on your comment, thanks.

config.yml

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /positions/positions.yaml

clients:
  - url: http://10.134.246.2:3100/loki/api/v1/push

scrape_configs:
- job_name: system
  static_configs:
  - targets:
      - localhost
    labels:
      job: varlogs
      __path__: /var/log/*log

Here is what I see in the Docker section of Unraid:

When I try and connect from Grafana now I get this error:

Data source connected, but no labels received. Verify that Loki and Promtail is configured properly.

[meddars]

I solved this by making sure that I was using the host IP in the config files, and starting Loki, then Promtail.