Is it possible to run Loki without errors?

[mnogokapel]

Friends, I’m trying to run Loki in Docker using these settings:
https://raw.githubusercontent.com/grafana/loki/v3.3.2/cmd/loki/loki-local-config.yaml
(Install Loki with Docker or Docker Compose | Grafana Loki documentation)

Here are my settings:

loki-config.yaml

auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9096
#  log_level: debug
  grpc_server_max_concurrent_streams: 1000

common:
  instance_addr: 127.0.0.1
  path_prefix: /tmp/loki
  storage:
    filesystem:
      chunks_directory: /tmp/loki/chunks
      rules_directory: /tmp/loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

query_range:
  results_cache:
    cache:
      embedded_cache:
        enabled: true
        max_size_mb: 100

#limits_config:
#  metric_aggregation_enabled: true

schema_config:
  configs:
    - from: 2020-10-24
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h

#pattern_ingester:
#  enabled: true
#  metric_aggregation:
#    enabled: true
#    loki_address: localhost:3100

ruler:
  alertmanager_url: http://localhost:9093

frontend:
  encoding: protobuf

And in the Loki logs, there are entries like this:

caller=scheduler_processor.go:175 component=querier org_id=fake traceID=30c8cf62a5e17c02 msg="error notifying scheduler about finished query" err=EOF addr=127.0.0.1:9096
caller=scheduler_processor.go:111 component=querier msg="error processing requests from scheduler" err="rpc error: code = Canceled desc = context canceled" addr=127.0.0.1:9096
caller=ratestore.go:109 msg="error getting ingester clients" err="empty ring"
caller=compactor.go:534 msg="failed to run compaction" err="context canceled"
caller=scheduler_processor.go:111 component=querier msg="error processing requests from scheduler" err="rpc error: code = Canceled desc = context canceled" addr=127.0.0.1:9096
caller=scheduler_processor.go:111 component=querier msg="error processing requests from scheduler" err="rpc error: code = Canceled desc = context canceled" addr=127.0.0.1:9096
caller=scheduler_processor.go:111 component=querier msg="error processing requests from scheduler" err="rpc error: code = Canceled desc = context canceled" addr=127.0.0.1:9096
caller=scheduler_processor.go:111 component=querier msg="error processing requests from scheduler" err="rpc error: code = Canceled desc = context canceled" addr=127.0.0.1:9096

---

If I uncomment:

limits_config:
  metric_aggregation_enabled: true

caller=scheduler_processor.go:111 component=querier msg="error processing requests from scheduler" err="rpc error: code = Canceled desc = context canceled" addr=127.0.0.1:9096
caller=ratestore.go:109 msg="error getting ingester clients" err="empty ring"
caller=scheduler_processor.go:111 component=querier msg="error processing requests from scheduler" err="rpc error: code = Canceled desc = context canceled" addr=127.0.0.1:9096
caller=scheduler_processor.go:111 component=querier msg="error processing requests from scheduler" err="rpc error: code = Canceled desc = context canceled" addr=127.0.0.1:9096
caller=scheduler_processor.go:111 component=querier msg="error processing requests from scheduler" err="rpc error: code = Canceled desc = context canceled" addr=127.0.0.1:9096
caller=compactor.go:534 msg="failed to run compaction" err="context canceled"
caller=scheduler_processor.go:111 component=querier msg="error processing requests from scheduler" err="rpc error: code = Canceled desc = context canceled" addr=127.0.0.1:9096

If I uncomment:

pattern_ingester:
  enabled: true
  metric_aggregation:
    enabled: true
    loki_address: localhost:3100

failed parsing config: /etc/loki/loki-config.yaml: yaml: unmarshal errors:
  line 44: field enabled not found in type aggregation.Config. Use `-config.expand-env=true` flag if you want to expand environment variables in your config file

---

docker compose

---
services:
  loki:
    image: grafana/loki:3.3.2
    container_name: loki
    restart: unless-stopped
    ports:
      - 3100:3100
    networks:
      - generic
    command: -config.file=/etc/loki/loki-config.yaml
    volumes:
      - /home/monitoring/loki/loki-config.yaml:/etc/loki/loki-config.yaml
      - /home/monitoring/loki:/data/loki
      
networks:
  generic:
    name: generic
    external: true

---

My question is: has anyone managed to run Loki without errors?
Even using the standard settings recommended by Grafana Labs, it’s impossible to run without errors. And if the configuration is more complex, does it become completely unfeasible?

I would greatly appreciate any help.

Best regards,
Alexander

[tonyswumac]

1. Try changing instance_addr to 0.0.0.0.
2. You need to expose gRPC port 9096, according to your configuration.
3. What is listening on 9093 from your alertmanager_url?

[mnogokapel]

Port forwarding didn’t help.

I made some changes:
I removed enabled: true in pattern_ingester, and now the container starts.

Here are my current settings with the changes:

loki-config.yaml

auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9096
  grpc_server_max_recv_msg_size: 20971520
  grpc_server_max_send_msg_size: 20971520

common:
  instance_addr: 0.0.0.0
  path_prefix: /tmp/loki
  storage:
    filesystem:
      chunks_directory: /tmp/loki/chunks
      rules_directory: /tmp/loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

query_range:
  results_cache:
    cache:
      embedded_cache:
        enabled: true
        max_size_mb: 100

limits_config:
  metric_aggregation_enabled: true

schema_config:
  configs:
    - from: 2020-10-24
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h

pattern_ingester:
  enabled: true
  metric_aggregation:
    loki_address: localhost:3100

frontend:
  encoding: protobuf

---

docker compose

---
services:
  loki:
    image: grafana/loki:3.3.2
    container_name: loki
    restart: unless-stopped
    user: root
    ports:
      - 3100:3100
      - 9096:9096
    networks:
      - generic
    command: "-config.file=/etc/loki/loki-config.yaml"
    volumes:
      - /home/monitoring/loki/loki-config.yaml:/etc/loki/loki-config.yaml
      - /home/monitoring/loki:/data/loki

networks:
  generic:
    name: generic
    external: true

Logs*

caller=scheduler_processor.go:111 component=querier msg="error processing requests from scheduler" err="rpc error: code = Canceled desc = context canceled" addr=0.0.0.0:9096
caller=scheduler_processor.go:111 component=querier msg="error processing requests from scheduler" err="rpc error: code = Canceled desc = context canceled" addr=0.0.0.0:9096
caller=ratestore.go:109 msg="error getting ingester clients" err="empty ring"
caller=scheduler_processor.go:111 component=querier msg="error processing requests from scheduler" err="rpc error: code = Canceled desc = context canceled" addr=0.0.0.0:9096
caller=scheduler_processor.go:111 component=querier msg="error processing requests from scheduler" err="rpc error: code = Canceled desc = context canceled" addr=0.0.0.0:9096
caller=scheduler_processor.go:111 component=querier msg="error processing requests from scheduler" err="rpc error: code = Canceled desc = context canceled" addr=0.0.0.0:9096
caller=scheduler_processor.go:111 component=querier msg="error processing requests from scheduler" err="rpc error: code = Canceled desc = context canceled" addr=0.0.0.0:9096

I’m starting to lose hope.

[tonyswumac]

Might I recommend you to change the gRPC port back to default 9095 and see if that fixes your problem?

If you change the default port you might have to also adjust your ring membership settings. I never changed the port, so I could be wrong.

[mnogokapel]

Each time upon launch, the number of errors is different, but they are always the same.
For example, such as:

caller=ratestore.go:109 msg="error getting ingester clients" err="empty ring"
caller=compactor.go:534 msg="failed to run compaction" err="context canceled"

There are often many like these:

caller=scheduler_processor.go:111 component=querier msg="error processing requests from scheduler" err="rpc error: code = Canceled desc = context canceled" addr=0.0.0.0:9097

---

I changed the port to 9097, but nothing has changed.

compose

ports:
  - 3100:3100
  - 9097:9097

loki-config.yaml

server:
  http_listen_port: 3100
  grpc_listen_port: 9097

logs

caller=scheduler_processor.go:111 component=querier msg="error processing requests from scheduler" err="rpc error: code = Canceled desc = context canceled" addr=0.0.0.0:9097

Maybe it’s not about the settings?

[tonyswumac]

Did you try 9095?

[mnogokapel]

Yes, I tried, but nothing changed.

caller=scheduler_processor.go:175 component=querier org_id=fake traceID=2923e13055eb2907 msg="error notifying scheduler about finished query" err=EOF addr=0.0.0.0:9095
caller=scheduler_processor.go:111 component=querier msg="error processing requests from scheduler" err="rpc error: code = Canceled desc = context canceled" addr=0.0.0.0:9095
caller=ratestore.go:109 msg="error getting ingester clients" err="empty ring"
caller=scheduler_processor.go:111 component=querier msg="error processing requests from scheduler" err="rpc error: code = Canceled desc = context canceled" addr=0.0.0.0:9095
caller=scheduler_processor.go:111 component=querier msg="error processing requests from scheduler" err="rpc error: code = Canceled desc = context canceled" addr=0.0.0.0:9095
caller=scheduler_processor.go:111 component=querier msg="error processing requests from scheduler" err="rpc error: code = Canceled desc = context canceled" addr=0.0.0.0:9095
caller=compactor.go:534 msg="failed to run compaction" err="context canceled"
caller=scheduler_processor.go:111 component=querier msg="error processing requests from scheduler" err="rpc error: code = Canceled desc = context canceled" addr=0.0.0.0:9095

I even tried your Loki settings:

Time until loki streams are searchable by querier

Unfortunately I am not able to replicate your issue, using your own configuration. Here is what I have: docker-compose.yml: --- networks: loki: services: loki: image: grafana/loki:3.1.0 command: "-config.file=/etc/loki/config.yaml -target=all" ports: - "3101:3101" - 7946 - 9095 volumes: - /root/loki/loki-config.yaml:/etc/loki/config.yaml - /root/loki/opt:/opt/loki:rw networks: loki: aliases: - loki grafana: i…

[tonyswumac]

I was able to get it to run without problem. Check your /ring endpoint to make sure a ring is formed.

Also, I forgot to point out, when you first start Loki empty ring is a benign error, because it’s empty until it recognizes itself as a member. If you don’t see it repeated it’s not a problem.

[mnogokapel]

I have a request for you: could you run Loki with my settings on your end to check if the errors will appear or not?
Try starting and stopping the compose several times. Sometimes, the number and type of errors vary.
For example, like this one:

caller=scheduler_processor.go:111 component=querier msg="error processing requests from scheduler" err="rpc error: code = Canceled desc = context canceled" addr=0.0.0.0:9095

Here are the settings:
loki-config.yaml

auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9095
  grpc_server_max_recv_msg_size: 20971520
  grpc_server_max_send_msg_size: 20971520

  grpc_server_max_concurrent_streams: 1000

common:
  instance_addr: 0.0.0.0
  path_prefix: /tmp/loki
  storage:
    filesystem:
      chunks_directory: /tmp/loki/chunks
      rules_directory: /tmp/loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

query_range:
  results_cache:
    cache:
      embedded_cache:
        enabled: true
        max_size_mb: 100

limits_config:
  metric_aggregation_enabled: true

schema_config:
  configs:
    - from: 2020-10-24
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h

pattern_ingester:
  enabled: true
  metric_aggregation:
    loki_address: localhost:3100

frontend:
  encoding: protobuf

compose

---
services:
  loki:
    image: grafana/loki:3.3.2
    container_name: loki
    restart: unless-stopped
    user: root
    ports:
      - 3100:3100
      - 9095:9095
    networks:
      - generic
    command: "-config.file=/etc/loki/loki-config.yaml"
    volumes:
      - /home/monitoring/loki/loki-config.yaml:/etc/loki/loki-config.yaml
      - /home/monitoring/loki:/data/loki

networks:
  generic:
    name: generic
    external: true

If there are no errors with my settings on your end, then the issue is not with the settings. (((