Logs disappear after restarting Loki on ECS with S3 backend

Grafana Loki
, , ,
1

Hi Grafana community,

I’m running Loki on AWS ECS with S3 as the storage backend. My setup has 2 Loki instances using the same config file.

Problem:

  • Logs that are ingested disappear from Grafana after restarting Loki.

Questions:

  1. Could this be because I need to mount EFS for shared storage?

  2. If I use EFS, can I mount the same EFS to both Loki instances safely?

  3. Is there a configuration change I need to ensure logs are flushed to S3 properly before restart?

Here’s my current setup:

auth_enabled: false

server:
http_listen_port: 3005

common:
path_prefix: /tmp/loki
ring:
kvstore:
store: memberlist

memberlist:
bind_port: 7946
join_members:

  • loki.ecs-namespace.local

schema_config:
configs:

  • from: 2020-10-24
    store: tsdb
    object_store: s3
    schema: v13
    index:
    prefix: logs_
    period: 24h

storage_config:
aws:
bucketnames: dev-monitoring-loki
region: us-west-1
s3forcepathstyle: false
tsdb_shipper:
active_index_directory: “/tmp/tsdb-index”
cache_location: “/tmp/tsdb-cache”

pattern_ingester:
lifecycler:
min_ready_duration: 1s

ingester:
lifecycler:
min_ready_duration: 1s

frontend:
scheduler_dns_lookup_period: 1s
address: 127.0.0.1

query_scheduler:

use_scheduler_ring: false

Thanks for any guidance!

2

I think you need to change your reference in schema config:

from: 2020-10-24
store: tsdb
object_store: aws # was s3
schema: v13
index:
prefix: logs_
period: 24h

If your logs are missing after restart then that means you are not actually storing logs in S3.

If you enable WAL on your ingester then you should consider a stateful directory for your ingester to store WAL logs. My recommendation would be to run your ingester in DAEMON mode, and then mount a directory from host into Loki container for WAL storage. That way WAL data are persisted between container restart.