Loki pod needs to be restarted sometimes to fetch old logs

Grafana Loki
1

I am running Grafana Loki in K8s ENV as a single pod and fetching data saved in the same using grafana dashboard.

Sometimes when I check for old data. Say one day before data, it doesen’t show me any logs/data. ( Gets a messages like: No logs/Data found).

When I try restarting the pod( Deleting a recreating the pod) and check again for the same date, I can see data populating without any issues.

Any Idea, Why this is happening and Is there any workaround for this.

Please help.

2

  1. Please share your loki configuration.

  2. Do you see any error message in the Loki pod when it fails to retrieve older logs?

3

Thank you for responding, Tonyswumac.

Here is the Loki config.

====
auth_enabled: false

server:
http_listen_port: 3100
grpc_listen_port: 9096

common:
instance_addr: 127.0.0.1
path_prefix: /tmp/loki
storage:
filesystem:
chunks_directory: /tmp/loki/chunks
rules_directory: /tmp/loki/rules
replication_factor: 1
ring:
kvstore:
store: inmemory

query_range:
results_cache:
cache:
embedded_cache:
enabled: true
max_size_mb: 100

schema_config:
configs:
- from: 2020-10-24
store: boltdb-shipper
object_store: filesystem
schema: v11
index:
prefix: index_
period: 24h

limits_config:
allow_structured_metadata: true
#EXTRA ADDED
ingestion_rate_strategy: local # Default: global
max_global_streams_per_user: 0
max_query_length: 0 # Default: 721h
max_query_parallelism: 32 # Old Default: 14
max_streams_per_user: 0 # Old Default: 10000

ruler:
alertmanager_url: http://localhost:9093

By default, Loki will send anonymous, but uniquely-identifiable usage and configuration

analytics to Grafana Labs. These statistics are sent to https://stats.grafana.org/

====

I couldn’t find any specific logs… Will check and post if I could find any when I face the same issue again.

4

Does the config looks fine…? Is there any thing to be changed in the config file to mitigate same issue from happening again…? Thanks

5

I updated Loki Version to latest version: 2.9.3 (Latest release as on 26-DEC-2023). Now waiting to see how it behaves now. Between any help or suggestions will be helpful.