Hello!
I set up Loki 2-3 months ago. Everything was working fine until last week, when I noticed that there was no logs history in Loki. I could only see logs from 4-6h back, but it’s not a strict rule.
Loki version: 3.4.1, but in the meantime, I upgraded to 3.4.6, hoping it will fix the issue.
What I tried:
I checked the loki configuration. The current one is:
auth_enabled: false
server:
http_listen_port: 3100
grpc_listen_port: 9095
log_level: info
log_format: logfmt
common:
storage:
filesystem:
chunks_directory: /loki/chunks
rules_directory: /loki/rules
replication_factor: 1
ring:
kvstore:
store: inmemory
schema_config:
configs:
- from: 2020-10-24
store: tsdb
object_store: s3
schema: v13
index:
prefix: index_
period: 24h
storage_config:
tsdb_shipper:
active_index_directory: /loki/index
cache_location: /loki/index_cache
aws:
# S3 configuration using EC2 instance IAM role
s3: s3://${AWS_REGION}/${LOKI_S3_BUCKET_NAME}
bucketnames: ${LOKI_S3_BUCKET_NAME}
region: ${AWS_REGION}
endpoint: ${LOKI_S3_ENDPOINT}
s3forcepathstyle: true
insecure: false
http_config:
idle_conn_timeout: 90s
response_header_timeout: 0s
insecure_skip_verify: false
ingester:
wal:
enabled: true
dir: /loki/wal
checkpoint_duration: 5m
flush_on_shutdown: true
replay_memory_ceiling: 4GB
max_chunk_age: 24h
chunk_idle_period: 1h
chunk_retain_period: 1h
querier:
max_concurrent: 16
compactor:
working_directory: /loki/compactor
compaction_interval: 10m
retention_enabled: false
limits_config:
reject_old_samples: false
ingestion_rate_mb: 100
ingestion_burst_size_mb: 200
max_query_lookback: 0s
table_manager:
retention_deletes_enabled: false
retention_period: 0s
Checked the S3 configuration. The EC2 machine that hosts Loki has the following permissions:
{
"Action": [
"s3:GetObject",
"s3:PutObject",
"s3:DeleteObject",
"s3:ListBucket",
"s3:GetBucketLocation",
"s3:GetBucketVersioning",
"s3:PutObjectAcl"
],
"Effect": "Allow",
"Resource": [
"arn:aws:s3:::my-loki-bucket",
"arn:aws:s3:::my-loki-bocket/*"
]
}
The variables from config resolves to:
AWS_REGION=eu-north-1
LOKI_S3_BUCKET_NAME=my-loki-bucket
LOKI_S3_ENDPOINT=my-loki-bucket.s3.eu-north-1.amazonaws.com
What I can see in Loki logs is a bunch of errors, when I try to fetch older logs:
loki | level=info ts=2025-09-26T12:57:04.88906553Z caller=metrics.go:237 component=querier org_id=fake traceID=42cc9349707222c9 latency=fast query="sum by (detected_level)(count_over_time({service_name=\"api\"}[20m]))" query_hash=71364292 query_type=metric range_type=range length=40m0s start_delta=146h57m4.887564114s end_delta=146h17m4.887564261s step=20m0s duration=74.34348ms status=200 limit=100 returned_lines=0 throughput=0B total_bytes=0B total_bytes_structured_metadata=0B lines_per_second=0 total_lines=0 post_filter_lines=0 total_entries=0 store_chunks_download_time=0s queue_time=207.801832ms splits=0 shards=0 query_referenced_structured_metadata=false pipeline_wrapper_filtered_lines=0 chunk_refs_fetch_time=74.308322ms cache_chunk_req=0 cache_chunk_hit=0 cache_chunk_bytes_stored=0 cache_chunk_bytes_fetched=0 cache_chunk_download_time=0s cache_index_req=0 cache_index_hit=0 cache_index_download_time=0s cache_stats_results_req=0 cache_stats_results_hit=0 cache_stats_results_download_time=0s cache_volume_results_req=0 cache_volume_results_hit=0 cache_volume_results_download_time=0s cache_result_req=0 cache_result_hit=0 cache_result_download_time=0s cache_result_query_length_served=0s cardinality_estimate=0 ingester_chunk_refs=0 ingester_chunk_downloaded=0 ingester_chunk_matches=0 ingester_requests=0 ingester_chunk_head_bytes=0B ingester_chunk_compressed_bytes=0B ingester_chunk_decompressed_bytes=0B ingester_post_filter_lines=0 congestion_control_latency=0s index_total_chunks=0 index_post_bloom_filter_chunks=0 index_bloom_filter_ratio=0.00 index_used_bloom_filters=false index_shard_resolver_duration=0s source=grafana-lokiexplore-app disable_pipeline_wrappers=false has_labelfilter_before_parser=false
loki | level=error ts=2025-09-26T12:57:04.890294582Z caller=index_set.go:308 table-name=index_20350 msg="sync failed, retrying it" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: B9W2QG0ABBTMMMHQ, host id: Bn2Fh5pXwHHK0zDSENFiI+huL8gD0uXSA7+d2GQWvXrve+VFyGDa11hjSjNPEEOf70+1ER3v4do="
loki | level=error ts=2025-09-26T12:57:04.903785513Z caller=index_set.go:308 table-name=index_20350 msg="sync failed, retrying it" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: B9W7VMHPZE4Q9FVN, host id: /zk2BIJJIYFoECJBM5yajxAuZCnmI7cTlj6q/ac9pcHxbERBii5LumdrSg387rty6SVNUnCNsmI="
loki | ts=2025-09-26T12:57:04.903818834Z caller=spanlogger.go:111 table-name=index_20350 user=fake caller=log.go:168 level=error msg="failed to initialize table, cleaning it up" table=index_20350 err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: B9W7VMHPZE4Q9FVN, host id: /zk2BIJJIYFoECJBM5yajxAuZCnmI7cTlj6q/ac9pcHxbERBii5LumdrSg387rty6SVNUnCNsmI="
loki | ts=2025-09-26T12:57:04.90384137Z caller=spanlogger.go:111 table-name=index_20350 user=fake caller=log.go:168 level=info msg="init index set" duration=26.906455ms success=false
loki | ts=2025-09-26T12:57:04.903877776Z caller=spanlogger.go:111 table-name=index_20350 user=fake caller=log.go:168 level=info msg="downloaded index set at query time" duration=26.906455ms
loki | ts=2025-09-26T12:57:04.90388636Z caller=spanlogger.go:111 table-name=index_20350 user=fake caller=log.go:168 level=error msg="failed to init user index set" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: B9W7VMHPZE4Q9FVN, host id: /zk2BIJJIYFoECJBM5yajxAuZCnmI7cTlj6q/ac9pcHxbERBii5LumdrSg387rty6SVNUnCNsmI="
loki | level=error ts=2025-09-26T12:57:04.903910666Z caller=table.go:370 table-name=index_20350 org_id=fake traceID=42cc9349707222c9 msg="index set has some problem, cleaning it up" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: B9W7VMHPZE4Q9FVN, host id: /zk2BIJJIYFoECJBM5yajxAuZCnmI7cTlj6q/ac9pcHxbERBii5LumdrSg387rty6SVNUnCNsmI="
loki | level=info ts=2025-09-26T12:57:04.90396909Z caller=table_manager.go:195 index-store=tsdb-2020-10-24 msg="get or create table" found=true table=index_20351 wait_for_lock=2.104µs
loki | level=error ts=2025-09-26T12:57:04.914952703Z caller=index_set.go:308 table-name=index_20351 user-id=fake msg="sync failed, retrying it" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: B9W8M3QT75V8V6MP, host id: 9hw2oJKVpCdFYWKYAmadd+NliwfyuExGKT0lqAE4B1RHVRkhCbm4s/L87LtS5fzGkYWTIVZ/4HQ="
loki | level=error ts=2025-09-26T12:57:04.92598284Z caller=index_set.go:308 table-name=index_20351 user-id=fake msg="sync failed, retrying it" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: B9WD34FENF46GN62, host id: 6XXbnOAiDDrNXNvx4lPwUGji5kODB7jTXj1oE/9hrx1Wmut6So4ZH4X05kb5cOAnPvjTSgXbdRI="
loki | ts=2025-09-26T12:57:04.926022221Z caller=spanlogger.go:111 table-name=index_20351 user-id=fake user=fake caller=log.go:168 level=error msg="failed to initialize table, cleaning it up" table=index_20351 err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: B9WD34FENF46GN62, host id: 6XXbnOAiDDrNXNvx4lPwUGji5kODB7jTXj1oE/9hrx1Wmut6So4ZH4X05kb5cOAnPvjTSgXbdRI="
loki | ts=2025-09-26T12:57:04.926045594Z caller=spanlogger.go:111 table-name=index_20351 user-id=fake user=fake caller=log.go:168 level=info msg="init index set" duration=21.919386ms success=false
loki | ts=2025-09-26T12:57:04.926063879Z caller=spanlogger.go:111 table-name=index_20351 user-id=fake user=fake caller=log.go:168 level=info msg="downloaded index set at query time" duration=21.919386ms
loki | ts=2025-09-26T12:57:04.926073247Z caller=spanlogger.go:111 table-name=index_20351 user-id=fake user=fake caller=log.go:168 level=error msg="failed to init user index set" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: B9WD34FENF46GN62, host id: 6XXbnOAiDDrNXNvx4lPwUGji5kODB7jTXj1oE/9hrx1Wmut6So4ZH4X05kb5cOAnPvjTSgXbdRI="
loki | level=error ts=2025-09-26T12:57:04.926095093Z caller=table.go:370 table-name=index_20351 org_id=fake traceID=42cc9349707222c9 msg="index set fake has some problem, cleaning it up" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: B9WD34FENF46GN62, host id: 6XXbnOAiDDrNXNvx4lPwUGji5kODB7jTXj1oE/9hrx1Wmut6So4ZH4X05kb5cOAnPvjTSgXbdRI="
loki | level=error ts=2025-09-26T12:57:04.939159248Z caller=index_set.go:308 table-name=index_20351 msg="sync failed, retrying it" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: B9WEA81G72SN5YFE, host id: E5CXFShlk7xlw99LlLmtHGHGFmxXOjQd3UxEzjpjfrhQ1VliD7iEVRPTNYArNffqdbGedBxxQ78="
loki | level=error ts=2025-09-26T12:57:04.949920638Z caller=index_set.go:308 table-name=index_20351 msg="sync failed, retrying it" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: B9W3CGYEJZ5AP99X, host id: sVBua25zXfYxSl1yTbWO2GUXIRIFaRqsWz+UIrWZuHZ7VBvHu/rAPh6C6ZgCP769VpAuo19+tOQ="
loki | ts=2025-09-26T12:57:04.94995715Z caller=spanlogger.go:111 table-name=index_20351 user=fake caller=log.go:168 level=error msg="failed to initialize table, cleaning it up" table=index_20351 err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: B9W3CGYEJZ5AP99X, host id: sVBua25zXfYxSl1yTbWO2GUXIRIFaRqsWz+UIrWZuHZ7VBvHu/rAPh6C6ZgCP769VpAuo19+tOQ="
loki | ts=2025-09-26T12:57:04.95001086Z caller=spanlogger.go:111 table-name=index_20351 user=fake caller=log.go:168 level=info msg="init index set" duration=23.749337ms success=false
loki | ts=2025-09-26T12:57:04.950029074Z caller=spanlogger.go:111 table-name=index_20351 user=fake caller=log.go:168 level=info msg="downloaded index set at query time" duration=23.749337ms
loki | ts=2025-09-26T12:57:04.950039261Z caller=spanlogger.go:111 table-name=index_20351 user=fake caller=log.go:168 level=error msg="failed to init user index set" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: B9W3CGYEJZ5AP99X, host id: sVBua25zXfYxSl1yTbWO2GUXIRIFaRqsWz+UIrWZuHZ7VBvHu/rAPh6C6ZgCP769VpAuo19+tOQ="
loki | level=error ts=2025-09-26T12:57:04.950059146Z caller=table.go:370 table-name=index_20351 org_id=fake traceID=42cc9349707222c9 msg="index set has some problem, cleaning it up" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: B9W3CGYEJZ5AP99X, host id: sVBua25zXfYxSl1yTbWO2GUXIRIFaRqsWz+UIrWZuHZ7VBvHu/rAPh6C6ZgCP769VpAuo19+tOQ="
Those logs lead me to the conclusion that maybe there is no such index in S3 “table=index_20351”. So I’ve checked it, and to my surprise it exists!
Is there any way to get older logs working? I can see them in S3, but Loki can’t fetch them. Is it know bug maybe?