Grafana is unable to connect to loki

Grafana Configuration
1
  • What Grafana version and what operating system are you using?
v11.2.0 (2a88694fd3)

Distributor ID: Ubuntu
Description:    Ubuntu 22.04.4 LTS
Release:        22.04
Codename:       jammy
  • What are you trying to achieve?
    Trying to connect Grafana to loki on the Data sources page for Loki
  • How are you trying to achieve it?
    By entering http://loki:3100, http://10.0.0.70:3100 or http://localhost:3100 in the Connection tab (URL)
  • What happened?
    Unable to connect with Loki. Please check the server logs for more details.
    If I go to the page http://loki:3100 on a web browser I get a 404 error.
  • What did you expect to happen?
    For it to connect successfully
  • Can you copy/paste the configuration(s) that you are having problems with?

docker-compose.yml

version: "3"
networks:
  loki:
services:
  loki:
    image: grafana/loki:2.4.0
    volumes:
      # Maps your host's loki configuration file to the expected path inside the container
      - /home/loki/loki1/loki-config.yml:/etc/loki/loki-config.yml
    ports:
      - "3100:3100"
    restart: unless-stopped
    command: -config.file=/etc/loki/loki-config.yml
    networks:
      - loki


  promtail:
    image: grafana/promtail:2.4.0
    volumes:
      - /var/log:/var/log
      # Maps your host's promtail configuration file to the expected path inside the container
      - /home/loki/promtail/promtail-config.yml:/etc/promtail/promtail-config.yml
    restart: unless-stopped
    command: -config.file=/etc/promtail/promtail-config.yml
    networks:
      - loki


  grafana:
    image: grafana/grafana:latest
    volumes:
      - /home/serveradmin/docker_volumes/grafana:/var/lib/grafana
    ports:
      - "3000:3000"
    restart: unless-stopped
    networks:
      - loki
    user: root:root

loki-config.yml

auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9096

common:
  path_prefix: /tmp/loki
  storage:
    filesystem:
      chunks_directory: /tmp/loki/chunks
      rules_directory: /tmp/loki/rules
  replication_factor: 1
  ring:
    instance_addr: 127.0.0.1
    kvstore:
      store: inmemory

schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h

ruler:
  alertmanager_url: http://localhost:9093

I may as well include promtail
promtail-config.yml

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:

# local machine logs

- job_name: local
  static_configs:
  - targets:
      - localhost
    labels:
      job: varlogs
      __path__: /var/log/*log
  • Did you receive any errors in the Grafana UI or in related logs? If so, please tell us exactly what they were.
    Unable to connect with Loki. Please check the server logs for more details.

docker logs loki_grafana_1 output:

logger=tsdb.loki endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=checkHealth pluginId=loki dsName=loki dsUID=cdxmlpmgt0pvkb uname=admin fromAlert=false t=2024-09-12T01:56:46.49556451Z level=error msg="Error received from Loki" duration=2.432519ms stage=databaseRequest statusCode=400 contentLength=65 start=1970-01-01T00:00:01Z end=1970-01-01T00:00:04Z step=1s query=vector(1)+vector(1) queryType=instant direction=backward maxLines=0 supportingQueryType=none lokiHost=loki:3100 lokiPath=/loki/api/v1/query status=error error="parse error at line 1, col 1: syntax error: unexpected IDENTIFIER" statusSource=downstream
logger=tsdb.loki endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth t=2024-09-12T01:56:46.495757751Z level=error msg="Loki health check failed" error="error from loki: parse error at line 1, col 1: syntax error: unexpected IDENTIFIER"
logger=context userId=1 orgId=1 uname=admin t=2024-09-12T01:56:46.495872518Z level=info msg="Request Completed" method=GET path=/api/datasources/uid/cdxmlpmgt0pvkb/health status=400 remote_addr=10.0.0.10 time_ms=6 duration=6.407487ms size=106 referer=http://10.0.0.70:3000/connections/datasources/edit/cdxmlpmgt0pvkb handler=/api/datasources/uid/:uid/health status_source=server
2

If the loki is in the same node, just change loki to the localhost in the promtail-config.yml

Before go to next line:
check in the shell: ping loki
If not replied, just use below

add the /etc/hosts with:
127.0.0.1 loki

Regards,
Fadjar

3

Not positive about what I should change in the promtail config

- job_name: local
  static_configs:
  - targets:
      - loki

Are you telling me to replace localhost with the host name “loki”?

Also, It looks like I do not have a hosts file. Should I create it?