Cannot explore the most recent log entries

Grafana Loki
1

We recently lifted our loki from 2.6.2 stepwise to 2.9.8. Our deployment is “simple scalable mode” as container instances in Azure with Azure storage account blob container as storage.
With 2.6.2 we had 1x read and 3x writer. Now with 2.9.8 we have 2x read, 1x backend and 3x write.
We also switched from boltdb to tsdb.

Problem is that when we Explore from Grafana we cannot see the logs until after approx 10-30 minutes.

I suspect it is because the query do no get result from logs cached at the ingesters before they are persisted to the blob container.

Here is our config:

auth_enabled: false

server:
  http_listen_port: 3100
  http_server_read_timeout: 130s
  http_server_write_timeout: 130s  
  grpc_listen_port: 9096
  grpc_server_max_recv_msg_size: 104857600
  grpc_server_max_send_msg_size: 104857600

common:
  path_prefix: /tmp/loki
  storage:
    azure:
      account_name: company9wnyv4rst
      container_name: loki
      use_managed_identity: true
      request_timeout: 0      
  replication_factor: 3
  ring:
    kvstore:
      store: memberlist
  compactor_grpc_address: backend-1-ci.company.net:9096

memberlist:
  advertise_port: 7946
  bind_port: 7946
  join_members:
    - read-1-ci.company.net:7946
    - read-2-ci.company.net:7946      
    - write-1-ci.company.net:7946
    - write-2-ci.company.net:7946      
    - write-3-ci.company.net:7946      

schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: azure
      schema: v11
      index:
        prefix: index_
        period: 24h  
    - from: 2023-05-01
      store: boltdb-shipper
      object_store: azure
      schema: v12
      index:
        prefix: index2_
        period: 24h
    - from: 2024-05-21
      store: tsdb
      object_store: azure
      schema: v13
      index:
        prefix: index3_        
        period: 24h

storage_config:
  tsdb_shipper:
    active_index_directory: /tmp/loki/tsdb-shipper-active
    cache_location: /tmp/loki/cache
    cache_ttl: 24h
    shared_store: azure  

compactor:
  working_directory: /mnt/data/compactor
  shared_store: azure
  compaction_interval: 5m
  retention_enabled: false
  delete_request_store: azure

querier:
  max_concurrent: 6

ingester_client:
  grpc_client_config:
    max_recv_msg_size: 104857600

ingester:
  chunk_encoding: snappy

analytics:
  reporting_enabled: false

limits_config:
  query_timeout: 2m

What can the problem be? Why did we not have it before?

2

I don’t see anything obviously wrong. There is a configuration to query ingester within certain time frame, but since you aren’t setting it it should be default value (3h).

Double check and make sure your reader can connect to writer on port 7946, and see if there is any log from either reader or writer that’s of interest.

3

Thank you, I will investigate as you suggest.

4

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.