Loki Logs not showing after some times, only last 3 hours logs are showing in grafana

Grafana Loki
1

Loki Logs not showing after some times, only last 3 hours logs are showing in grafana. im using loki 3.5.2 in k8 with external minio bucket as storage. below is my configurations. can anybody help with the issue. i require a log retention for 30days.

server:
    http_listen_port: 3100
    grpc_listen_port: 9095
    http_server_read_timeout: 600s
    http_server_write_timeout: 600s
    
  # -- Limits config
  limits_config:
    reject_old_samples: true
    reject_old_samples_max_age: 168h
    max_cache_freshness_per_query: 10m
    split_queries_by_interval: 15m
    query_timeout: 300s
    max_query_lookback: 720h
    retention_period: 720h
    volume_enabled: true
    
  runtimeConfig: {}
  commonConfig:
    path_prefix: /var/loki
    replication_factor: 3
    compactor_address: '{{ include "loki.compactorAddress" . }}'
    
  storage:
    bucketNames:
       chunks: "xyz-logging"
       ruler: "xyz-logging"
       admin: "xyz-logging"
    type: s3
    s3:
      endpoint: "https://storage-api.xyzzyz.com"
      secretAccessKey: "<>"
      accessKeyId: "<>"
      signatureVersion: "v4"
      s3ForcePathStyle: true
      insecure: false
      http_config: {}
      backoff_config: {}
      disable_dualstack: false
    filesystem:
      chunks_directory: /var/loki/chunks
      rules_directory: /var/loki/rules
      admin_api_directory: /var/loki/admin

  schemaConfig:
    configs:
      - from: "2024-04-01"
        index:
          prefix: index_
          period: 24h
        store: tsdb
        object_store: s3
        schema: v13          
        
  storage_config:
      tsdb_shipper:
          active_index_directory: /data/index
          cache_location: /data/index_cache
      s3:
        bucketnames: xyz-logging
        s3forcepathstyle: true
  
  useTestSchema: false
  testSchemaConfig:
    configs:
      - from: 2024-04-01
        store: tsdb
        object_store: '{{ include "loki.testSchemaObjectStore" . }}'
        schema: v13
        index:
          prefix: index_
          period: 24h

  rulerConfig:
    wal:
      dir: /var/loki/ruler-wal
  structuredConfig: {}
  query_scheduler: {}
  compactor:
    working_directory: /var/loki/data/retention
    compaction_interval: 10m
    retention_enabled: true
    retention_delete_delay: 2h
    retention_delete_worker_count: 30
    delete_request_store: s3    
 
  pattern_ingester:
    enabled: false
  query_range: {}
  querier:
    max_concurrent: 4
  ingester:
     chunk_encoding: snappy
2

Hello @mailsaashiq ,

You can follow these steps to resolve your issue:
Step: 1 Change your Loki YAML file

index_gateway:
  mode: simple
  shared_store: s3

tsdb_shipper:
  active_index_directory: /data/index
  cache_location: /data/index_cache

compactor:
  working_directory: /var/loki/data/retention

chunk_idle_period: 24h

query_ingester_within: 25h

Step: 2 Update Grafana to Allow 30-Day Queries
Adjust the time range in the Grafana Explore panel as shown below:

image
image1356×580 78.6 KB

Step: 3 Apply the changes and restart Loki.

Hello @jangaraj, @yosiasz,

Could you please review my solution? If it is correct, kindly mark it as the accepted solution.

Thank you!

3

hi, the issue got resolved with your setting ( but didn’t find this settings- shared_store: s3 ), now the resource utilizations is reduced and old logs are also showing. here is my current setting.


  limits_config:
    retention_period: 744h
    volume_enabled: true
   
  storage_config:
      tsdb_shipper:
          active_index_directory: /data/index
          cache_location: /data/index_cache

  compactor:
    retention_enabled: true
    delete_request_store: s3   

  index_gateway:
    mode: simple
    
  querier:
    max_concurrent: 4
    query_ingesters_within: 25h
    
  ingester:
     chunk_encoding: snappy
     chunk_idle_period: 24h

i am currently facing another problem, the loki-chunks-cache is increasing memory daily. i have used default settings for the chunk-cache in the values.yaml. any options to decrease the high memory usage. current replica is 1