Loki not showing logs older than 2hrs on grafana dashboard

[devdattaj]

What happened?
When we try time series visualization of Logs when Loki as Datasource, we get

graph and logs only upto last 2 hrs of timespan. If we increase the time range or use older time its not getting the data on dashboard. Also its not deleting any data from storage bucket older than retention period.

What did you expect to happen?
We should able to get the data on dashboard for atleast 24 hrs or for time whatever our retention period is. It should delete the older data from storage and retrieve all available data.

Did this work before?
Yes this worked before and we were getting all the logs when using filesystem as storage in loki-config.yaml but when we started using digitalocean space storage buckets its started showing only recent 2 hrs of data. Also using the limits_config as given below the log retrival work for one time.
max_query_series: 50000
max_entries_limit_per_query: 100000

How do we reproduce it?:

1. Use Alloy to send logs to Loki and Loki as Data source in Grafana
2. First use filesystem as log storage option and then switch to storage space buckets.

Is the bug inside a dashboard panel?
No, this bug is not inside dashboard panel.

Environment (with versions):

• Grafana version: Version 12.0.2
• Operating system: Ubuntu 22.04
• Browser: Chrome Version 138.0.7204.97 (Official Build) (64-bit)
• Datasource version(s) (prometheus, graphite, etc.): Loki “version”:“3.5.1”
• Plugins: Loki

Configuration information:
Alloy agent, Loki and Grafana all are installed on single ubuntu server. Alloy fetch the logs and forward it to Loki. Loki store the log data to digitalocean storage buckets and Grafana is configured with Loki as data source manually from UI. The Loki config file is as given below.
auth_enabled: false

server:
http_listen_port: 3100
grpc_listen_port: 9096
log_level: info
grpc_server_max_concurrent_streams: 2000

common:
instance_addr: 127.0.0.1
path_prefix: /tmp/loki
storage:
#filesystem:
# chunks_directory: /tmp/loki/chunks
# rules_directory: /tmp/loki/rules
s3:
endpoint: https://xxxxxx.blr1.digitaloceanspaces.com
bucketnames: Loki
region: blr1
access_key_id: DxxxxxxxxxxxxxR
secret_access_key: bxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxY
s3forcepathstyle: true
replication_factor: 1
ring:
kvstore:
store: inmemory

query_range:
results_cache:
cache:
embedded_cache:
enabled: true
max_size_mb: 100

limits_config:
metric_aggregation_enabled: true
allow_structured_metadata: true
volume_enabled: true
retention_period: 24h # 24h
max_query_series: 50000
max_entries_limit_per_query: 100000

schema_config:
configs:
- from: 2025-07-04
store: tsdb
//#### object_store: filesystem
object_store: s3
schema: v13
index:
prefix: index_
period: 24h

pattern_ingester:
enabled: true
metric_aggregation:
loki_address: localhost:3100

ruler:
enable_alertmanager_discovery: true
enable_api: true

frontend:
encoding: protobuf

compactor:
working_directory: /tmp/loki/retention
delete_request_store: s3
retention_enabled: true
retention_delete_delay: 2h

[tonyswumac]

If you cannot see logs older than 2 hours then that means one of two things:

1. Your chunk files are not being sent to the S3 bucket.
2. Your Loki instance does not have permission to read from S3 bucket.

I would check and see if there is any error log from your Loki instance. Also, how many Loki instances are you running?

[devdattaj]

1. Chunk files are geting forwarded to S3 bucket and stored in s3 bucket.
2. The access key used in configuration have all permissions on bucket.
3. I am using single instance of Loki, installed on ubuntu vm.
4. Following are the error logs from loki service.
   Jul 10 19:28:18 docker-ubuntu-s-2vcpu-4gb-blr1-01 loki[404454]: level=info ts=2025-07-10T19:28:17.975942703Z caller=metrics.go:237 component=querier org_id=fake traceID=42062fa976cf5163 latency=fast query=“sum by (level,detected_level)(count_over_time({instance="docker-ubuntu-s-2vcpu-4gb-blr1-01"} |= "" | drop error[1m]))” query_hash=3462061732 query_type=metric range_type=range length=59m0s start_delta=5h28m17.975925065s end_delta=4h29m17.975925235s step=1m0s duration=14.333493ms status=200 limit=100 returned_lines=0 throughput=0B total_bytes=0B total_bytes_structured_metadata=0B lines_per_second=0 total_lines=0 post_filter_lines=0 total_entries=0 store_chunks_download_time=0s queue_time=47.461912ms splits=0 shards=0 query_referenced_structured_metadata=false pipeline_wrapper_filtered_lines=0 chunk_refs_fetch_time=14.259615ms cache_chunk_req=0 cache_chunk_hit=0 cache_chunk_bytes_stored=0 cache_chunk_bytes_fetched=0 cache_chunk_download_time=0s cache_index_req=0 cache_index_hit=0 cache_index_download_time=0s cache_stats_results_req=0 cache_stats_results_hit=0 cache_stats_results_download_time=0s cache_volume_results_req=0 cache_volume_results_hit=0 cache_volume_results_download_time=0s cache_result_req=0 cache_result_hit=0 cache_result_download_time=0s cache_result_query_length_served=0s cardinality_estimate=0 ingester_chunk_refs=0 ingester_chunk_downloaded=0 ingester_chunk_matches=0 ingester_requests=0 ingester_chunk_head_bytes=0B ingester_chunk_compressed_bytes=0B ingester_chunk_decompressed_bytes=0B ingester_post_filter_lines=0 congestion_control_latency=0s index_total_chunks=0 index_post_bloom_filter_chunks=0 index_bloom_filter_ratio=0.00 index_used_bloom_filters=false index_shard_resolver_duration=0s source=logvolhist disable_pipeline_wrappers=false has_labelfilter_before_parser=false
   Jul 10 19:28:18 docker-ubuntu-s-2vcpu-4gb-blr1-01 loki[404454]: level=info ts=2025-07-10T19:28:17.977284205Z caller=metrics.go:237 component=frontend org_id=fake traceID=42062fa976cf5163 latency=fast query=“sum by (level, detected_level) (count_over_time({instance="docker-ubuntu-s-2vcpu-4gb-blr1-01"} |= `` | drop error[1m]))” query_hash=772405044 query_type=metric range_type=range length=6h0m17.004s start_delta=6h0m17.977255712s end_delta=973.255995ms step=1m0s duration=133.474703ms status=200 limit=100 returned_lines=0 throughput=6.3MB total_bytes=843kB total_bytes_structured_metadata=86kB lines_per_second=39490 total_lines=5271 post_filter_lines=5271 total_entries=5 store_chunks_download_time=642.533µs queue_time=296.660224ms splits=7 shards=7 query_referenced_structured_metadata=true pipeline_wrapper_filtered_lines=0 chunk_refs_fetch_time=143.790201ms cache_chunk_req=0 cache_chunk_hit=0 cache_chunk_bytes_stored=0 cache_chunk_bytes_fetched=0 cache_chunk_download_time=0s cache_index_req=0 cache_index_hit=0 cache_index_download_time=0s cache_stats_results_req=6 cache_stats_results_hit=6 cache_stats_results_download_time=189.082µs cache_volume_results_req=0 cache_volume_results_hit=0 cache_volume_results_download_time=0s cache_result_req=0 cache_result_hit=0 cache_result_download_time=0s cache_result_query_length_served=0s cardinality_estimate=0 ingester_chunk_refs=3 ingester_chunk_downloaded=3 ingester_chunk_matches=16 ingester_requests=4 ingester_chunk_head_bytes=404kB ingester_chunk_compressed_bytes=66kB ingester_chunk_decompressed_bytes=439kB ingester_post_filter_lines=5271 congestion_control_latency=0s index_total_chunks=0 index_post_bloom_filter_chunks=0 index_bloom_filter_ratio=0.00 index_used_bloom_filters=false index_shard_resolver_duration=0s source=logvolhist disable_pipeline_wrappers=false has_labelfilter_before_parser=false
   Jul 10 19:28:18 docker-ubuntu-s-2vcpu-4gb-blr1-01 loki[404454]: level=error ts=2025-07-10T19:28:18.003392089Z caller=index_set.go:308 table-name=index_20279 user-id=fake msg=“sync failed, retrying it” err="NoSuchKey: \n\tstatus code: 404, request id: tx000002f82a04a93c92910-0068701451-11f95135-blr1a, host id: "
   Jul 10 19:28:18 docker-ubuntu-s-2vcpu-4gb-blr1-01 loki[404454]: level=error ts=2025-07-10T19:28:18.007326039Z caller=index_set.go:308 table-name=index_20279 user-id=fake msg=“sync failed, retrying it” err="NoSuchKey: \n\tstatus code: 404, request id: tx00000ab98c1664521d6af-0068701451-10b19b1a-blr1a, host id: "
   Jul 10 19:28:18 docker-ubuntu-s-2vcpu-4gb-blr1-01 loki[404454]: ts=2025-07-10T19:28:18.007362758Z caller=spanlogger.go:116 table-name=index_20279 user-id=fake user=fake level=error msg=“failed to initialize table, cleaning it up” table=index_20279 err="NoSuchKey: \n\tstatus code: 404, request id: tx00000ab98c1664521d6af-0068701451-10b19b1a-blr1a, host id: "
   Jul 10 19:28:18 docker-ubuntu-s-2vcpu-4gb-blr1-01 loki[404454]: ts=2025-07-10T19:28:18.007833192Z caller=spanlogger.go:116 table-name=index_20279 user-id=fake user=fake level=error msg=“failed to init user index set” err="NoSuchKey: \n\tstatus code: 404, request id: tx00000ab98c1664521d6af-0068701451-10b19b1a-blr1a, host id: "
   Jul 10 19:28:18 docker-ubuntu-s-2vcpu-4gb-blr1-01 loki[404454]: level=error ts=2025-07-10T19:28:18.007847592Z caller=table.go:370 table-name=index_20279 org_id=fake traceID=3059c64d47f593c8 msg=“index set fake has some problem, cleaning it up” err="NoSuchKey: \n\tstatus code: 404, request id: tx00000ab98c1664521d6af-0068701451-10b19b1a-blr1a, host id: "
   Jul 10 19:28:18 docker-ubuntu-s-2vcpu-4gb-blr1-01 loki[404454]: level=error ts=2025-07-10T19:28:18.011043047Z caller=index_set.go:308 table-name=index_20279 msg=“sync failed, retrying it” err="NoSuchKey: \n\tstatus code: 404, request id: tx00000ce893f416f0e31c6-0068701451-11f95199-blr1a, host id: "
   Jul 10 19:28:18 docker-ubuntu-s-2vcpu-4gb-blr1-01 loki[404454]: level=error ts=2025-07-10T19:28:18.01486211Z caller=index_set.go:308 table-name=index_20279 msg=“sync failed, retrying it” err="NoSuchKey: \n\tstatus code: 404, request id: tx000004fbc0235417b5c59-0068701451-11bdf37a-blr1a, host id: "
   Jul 10 19:28:18 docker-ubuntu-s-2vcpu-4gb-blr1-01 loki[404454]: ts=2025-07-10T19:28:18.014919026Z caller=spanlogger.go:116 table-name=index_20279 user=fake level=error msg=“failed to initialize table, cleaning it up” table=index_20279 err="NoSuchKey: \n\tstatus code: 404, request id: tx000004fbc0235417b5c59-0068701451-11bdf37a-blr1a, host id: "
   Jul 10 19:28:18 docker-ubuntu-s-2vcpu-4gb-blr1-01 loki[404454]: ts=2025-07-10T19:28:18.014997643Z caller=spanlogger.go:116 table-name=index_20279 user=fake level=error msg=“failed to init user index set” err="NoSuchKey: \n\tstatus code: 404, request id: tx000004fbc0235417b5c59-0068701451-11bdf37a-blr1a, host id: "
   Jul 10 19:28:18 docker-ubuntu-s-2vcpu-4gb-blr1-01 loki[404454]: level=error ts=2025-07-10T19:28:18.015017854Z caller=table.go:370 table-name=index_20279 org_id=fake traceID=3059c64d47f593c8 msg=“index set has some problem, cleaning it up” err="NoSuchKey: \n\tstatus code: 404, request id: tx000004fbc0235417b5c59-0068701451-11bdf37a-blr1a, host id: "