Failed to read cluster seed file : "open loki_cluster_seed.json: no such file or directory"

I am trying to deploy this grafana / loki stack :

Docker-compose.yml :

---
version: "3.7"

x-loki: &defaultloki
  image: grafana/loki:main
  volumes:
    - ./monitor/loki/:/opt/loki/
  networks:
    - loki

services:
  grafana:
    container_name: grafana
    image: grafana/grafana:latest
    restart: unless-stopped
    user: "${UID:-1001}:${GID:-1001}"
    volumes:
      - ./monitor/grafana/grafana.ini:/etc/grafana/grafana.ini
      - grafana_data:/var/lib/grafana
    networks:
      - loki
    ports:
      - 3000:3000
    environment:
      - GF_SMTP_ENABLED=true
      - GF_SMTP_HOST=mail.jurihub.fr:587
      - GF_SMTP_USER=grafana@jurihub.fr
      - GF_SMTP_PASSWORD=${GRAFANA_SMTP_PASSWORD}
      - GF_SMTP_FROM_ADDRESS=grafana@jurihub.fr
      - GF_DATABASE_TYPE=mysql
      - GF_DATABASE_HOST=grafana_mysql:3306
      - GF_DATABASE_NAME=grafana
      - GF_DATABASE_USER=grafana
      - GF_DATABASE_PASSWORD=${GRAFANA_MYSQL_PASSWORD}
    depends_on:
      - grafana_mysql

  grafana_mysql:
    container_name: grafana_mysql
    restart: unless-stopped
    image: mysql:8.0
    user: "${UID:-1001}:${GID:-1001}"
    volumes:
      - grafana_mysql:/var/lib/mysql
    networks:
      - loki
    environment:
      MYSQL_ROOT_PASSWORD: ${GRAFANA_MYSQL_ROOT_PASSWORD}
      MYSQL_DATABASE: grafana
      MYSQL_USER: grafana
      MYSQL_PASSWORD: ${GRAFANA_MYSQL_PASSWORD}

  # Read-path nodes.
  loki-read1:
    <<: *defaultloki
    command: "-config.expand-env=true -config.file=/opt/loki/loki.yml -target=read -querier.frontend-address=loki-frontend"
    ports:
    - 3100:3100

  loki-read2:
    <<: *defaultloki
    command: "-config.expand-env=true -config.file=/opt/loki/loki.yml -target=read -querier.frontend-address=loki-frontend"
    ports:
    - 3101:3100

  loki-read3:
    <<: *defaultloki
    command: "-config.expand-env=true -config.file=/opt/loki/loki.yml -target=read -querier.frontend-address=loki-frontend"
    ports:
    - 3102:3100

  # Write-path nodes.
  loki-write1:
    <<: *defaultloki
    command: "-config.expand-env=true -config.file=/opt/loki/loki.yml -target=write -ingester.lifecycler.ID=write1"
    ports:
    - 3200:3100

  loki-write2:
    <<: *defaultloki
    command: "-config.expand-env=true -config.file=/opt/loki/loki.yml -target=write -ingester.lifecycler.ID=write2"
    ports:
    - 3201:3100

  loki-write3:
    <<: *defaultloki
    command: "-config.expand-env=true -config.file=/opt/loki/loki.yml -target=write -ingester.lifecycler.ID=write3"
    ports:
    - 3202:3100

  loki-frontend:
    <<: *defaultloki
    command: "-config.expand-env=true -config.file=/opt/loki/loki.yml -target=query-frontend"
    ports:
      - 3301:3100

# Network
networks:
  loki:

volumes:
  grafana_mysql:
  grafana_data:

./monitor/loki/loki.yml :

auth_enabled: false

server:
  log_level: debug
  http_listen_port: 3100
  http_server_read_timeout: 300s # allow longer time span queries
  http_server_write_timeout: 300s # allow longer time span queries

memberlist:
  join_members:
  - loki-write1
  - loki-write2
  - loki-write3

ingester:
  wal:
    enabled: false
  lifecycler:
    unregister_on_shutdown: false
    join_after: 60s
    observe_period: 5s
    final_sleep: 0s

schema_config:
  configs:
  - from: 2020-08-01
    store: boltdb-shipper
    object_store: filesystem
    schema: v11
    index:
      prefix: index_
      period: 24h

ruler:
  storage:
    type: local
    local:
      directory: /tmp/loki/rules

limits_config:
  unordered_writes: true

query_range:
  cache_results: false

storage_config:
  boltdb_shipper:
    cache_ttl: 24h         # Can be increased for faster performance over longer query periods, uses more disk space
    shared_store: filesystem
    
common:
  path_prefix: /tmp/loki
  ring:
    kvstore:
      store: memberlist
  compactor_address: 0.0.0.0

Containers seem to connect with each other, but my query-frontend cannot read the cluster config, and therefore cannot communicate with readers (as far as I understand). There isn’t any line about this in loki documentation, and I already passed a lot of time on this…

The writers logs are :

...
ts=2023-04-13T16:15:11.9263426Z caller=memberlist_logger.go:74 level=debug msg="Initiating push/pull sync with:  192.168.32.8:7946"
ts=2023-04-13T16:15:11.9264365Z caller=memberlist_logger.go:74 level=debug msg="Stream connection from=192.168.32.8:39622"
ts=2023-04-13T16:15:11.9281191Z caller=memberlist_logger.go:74 level=debug msg="Stream connection from=192.168.32.5:56550"
ts=2023-04-13T16:15:11.9281737Z caller=memberlist_logger.go:74 level=debug msg="Stream connection from=192.168.32.3:43104"
ts=2023-04-13T16:15:11.9282043Z caller=memberlist_logger.go:74 level=debug msg="Initiating push/pull sync with:  192.168.32.4:7946"
ts=2023-04-13T16:15:11.9302569Z caller=memberlist_logger.go:74 level=debug msg="Stream connection from=192.168.32.3:43138"
...
level=info ts=2023-04-13T16:15:12.6244388Z caller=memberlist_client.go:595 msg="joining memberlist cluster succeeded" reached_nodes=3 elapsed_time=10.2433ms
ts=2023-04-13T16:15:47.3962164Z caller=memberlist_logger.go:74 level=debug msg="Initiating push/pull sync with: d3b28b760ea5-80264658 192.168.32.5:7946"
level=debug ts=2023-04-13T16:15:52.6204249Z caller=reporter.go:202 msg="failed to read cluster seed file" err="open loki_cluster_seed.json: no such file or directory"
ts=2023-04-13T16:16:08.2724812Z caller=memberlist_logger.go:74 level=debug msg="Stream connection from=192.168.32.9:32942"
ts=2023-04-13T16:16:09.7169489Z caller=memberlist_logger.go:74 level=debug msg="Stream connection from=192.168.32.3:44154"

The query-frontend logs are (readers logs are similar) :

...
level=info ts=2023-04-13T16:15:10.2026513Z caller=module_service.go:82 msg=initialising module=query-frontend
level=info ts=2023-04-13T16:15:10.2027464Z caller=loki.go:502 msg="Loki started"
level=debug ts=2023-04-13T16:15:11.8768536Z caller=reporter.go:202 msg="failed to read cluster seed file" err="open loki_cluster_seed.json: no such file or directory"

I tried to launch containers as root on my local computer but it wasn’t working neither.

Any help appreciated,
thanks !

If you are using local filesystem try a monolithic deployment. If you want to go with scalable deployment you should consider using an object storage such as S3.

1 Like

Thanks a lot !
I used Minio and I can now connect grafana to my loki “scalable”.
The problem I face now is I do not figure how to use a query-frontend on top of the readers.

Note that I started from : DylanGuedes/ssd-playground (github.com)

From my previous config, I added :

frontend_worker:
  frontend_address: loki-read1:3100

frontend:
  log_queries_longer_than: 1h
  max_outstanding_per_tenant: 2048
  # downstream_url: http://loki-frontend:3100 (if set, infinite query redirection)
  compress_responses: true

And in Grafana, I try to connect to : http://loki-frontend:3100

Request is received and it tries to resolve it (loki-frontend logs):

level=debug ts=2023-04-14T09:37:07.4250215Z caller=logging.go:76 traceID=4754b110e6f4d34a orgID=fake msg="GET /loki/api/v1/labels?start=1681464397381000000&end=1681464997381000000 (499) 29.9996057s"

In queriers :

level=error ts=2023-04-14T10:04:03.2266616Z caller=frontend_processor.go:63 msg="error contacting frontend" address=192.168.160.6:3100 err="rpc error: code = Unavailable desc = connection error: desc = \"error reading server preface: http2: frame too large\""

But “context is canceled” after the 30s default timeout.

I’m sorry, I am quite new on this subject, but single instance Loki is becoming too slow for us. Thanks for your help again.

I suspect it’s a problem with what you set for frontend_address.

There are two ways to configure query frontend, you can either configure query frontend to “push” queries to querier, or you can configure queriers to “pull” from query frontend. See Query Frontend | Grafana Loki documentation for more details. Generally I believe generally pull method is preferred.

To configure querier to pull from frontend, you configure frontend_worker.frontend_address for the querier to pull from, so in your configuration loki-read1:3100 needs to be reachable from queriers. From outside, you’d want to make sure incoming traffic for read traffic gets routed to query frontend.

Thank you again, this is helping me a lot.
I should now have everything set up and working.

But I have another error on which I can not find any information on. Whenever my queriers try to communicate with query-frontend, I get this error :

level=error ts=2023-04-17T13:00:05.2829843Z caller=frontend_processor.go:63 msg="error contacting frontend" address=192.168.192.6:3100 err="rpc error: code = Unavailable desc = connection error: desc = \"error reading server preface: http2: frame too large\""

Seems like request payload is being too heavy. I tried to increase max_recv_msg_size everywhere I could but I am still stuck…

Thank you for your time, I hope it is the last time I write in this topic.

frontend_worker.frontend_address should be configured to go to the gRPC port, which is 9095 by default. Sounds like you configured it to go to the Loki HTTP port instead, which would be incorrect.

1 Like