Loki logs not appearing, with error message log volume not configured

Grafana Loki
1

So we have had a deployment of Grafana on our web server for some time now. Just about a week ago, the logs on the grafana dashboard stopped appearing, with the following error message;
Log volume has not been configured.

Instructions to enable volume in the Loki config:

limits_config:  volume_enabled: true

Below is our configuration for loki:

auth_enabled: false

server:
  http_listen_address: 0.0.0.0
  grpc_listen_address: 0.0.0.0
  http_listen_port: 3100
  grpc_listen_port: 9095
  log_level: info

common:
  path_prefix: /loki
  compactor_address: http://loki-backend:3100
  replication_factor: 3

storage_config:
  aws:
    endpoint: ${AWS_STORAGE_ENDPOINT}
    insecure: true
    bucketnames: ${LOKI_DATA_BUCKET_NAME}
    access_key_id: ${AWS_ACCESS_KEY_ID}
    secret_access_key: ${AWS_SECRET_ACCESS_KEY}
    s3forcepathstyle: true

memberlist:
  join_members: ["loki-read", "loki-write", "loki-backend"]
  dead_node_reclaim_time: 30s
  gossip_to_dead_nodes_time: 15s
  left_ingesters_timeout: 30s
  bind_addr: ['0.0.0.0']
  bind_port: 7946
  gossip_interval: 2s

ingester:
  lifecycler:
    join_after: 10s
    observe_period: 5s
    ring:
      replication_factor: 3
      kvstore:
        store: memberlist
    final_sleep: 0s
  chunk_idle_period: 10m
  autoforget_unhealthy: true
  wal:
    enabled: true
    dir: /loki/wal
  chunk_retain_period: 30s
  chunk_encoding: snappy
  chunk_target_size: 1.572864e+06
  chunk_block_size: 262144
  flush_op_timeout: 10s

schema_config:
  configs:
  - from: 2020-08-01
    store: boltdb-shipper
    object_store: s3
    schema: v11
    index:
      prefix: index_
      period: 24h
  - from: 2023-07-11
    store: tsdb
    object_store: s3
    schema: v12
    index:
      prefix: index_
      period: 24h
  - from: 2024-01-10
    store: tsdb
    object_store: s3
    schema: v12
    index:
      prefix: index_
      period: 24h


limits_config:
  max_cache_freshness_per_query: '10m'
  ingestion_rate_mb: 10
  ingestion_burst_size_mb: 20
  # parallelize queries in 15min intervals
  split_queries_by_interval: 15m
  per_stream_rate_limit: 5MB
  per_stream_rate_limit_burst: 20MB
  volume_enabled: true

table_manager:
  retention_deletes_enabled: true
  retention_period: 336h

query_range:
  # make queries more cache-able by aligning them with their step intervals
  align_queries_with_step: true
  max_retries: 5
  parallelise_shardable_queries: true
  cache_results: true

frontend:
  log_queries_longer_than: 5s
  compress_responses: true
  max_outstanding_per_tenant: 2048

query_scheduler:
  max_outstanding_requests_per_tenant: 1024

querier:
  query_ingesters_within: 2h

compactor:
  working_directory: /tmp/compactor

I am trying to figure out what could have possibly gone wrong.

1 Like
2

volume_enabled defaults to true, you shouldn’t have to set it explicitly, and I doubt that’s your actual problem. I would try:

  1. Send an API call directly to your Loki endpoint using the /loki/api/v1/index/volume API. this should tell you whether volumes really is your problem.
  2. Check any gateway or proxy between Loki and Grafana, if you have any.
  3. What changed when the problem showed up? Was there any upgrade? configuration change?
3

I was told there was an upgrade and that triggered the errors.

4

Hi @ghenyo,
Just update your Loki to the latest version; it will be fixed after that.

5

I did that, nothing was resolved

6

It seems there may be a problem from the API endpoint, I am still looking into that
It returns a 404 error for 2 loki instances and a parse error for one other loki instance

7

Any updates on this? I use grafana/loki:3.4.4, and it still has this problem.

8

I have resolved the issue, the problem was with the ingester nodes, what was expected was more than what was actually available.

9

It seems that this issue can have multiple causes. For me, it turned out that it was rooted in an invalid URL in my Grafana data source configuration for Loki. It´s a pity that Grafana cannot give a better error message in its UI.