Labels lost on docker redeployment

I am testing Loki in a monolithic deployment using Docker and with S3 as object storage. Everything is working fine and I can see the chunks being uploaded to S3 in the fake/ directory. However, when I delete the container and deploy a new container using the same S3 bucket, Loki seems to not be able to restore my labels. I don’t get any labels back from GET /loki/api/v1/labels. Am I missing something?

loki-config.yml

auth_enabled: false

server:
  http_listen_port: 3100

common:
  path_prefix: /loki
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

query_range:
  results_cache:
    cache:
      embedded_cache:
        enabled: true
        max_size_mb: 100

schema_config:
  configs:
    - from: 2020-05-15
      store: boltdb-shipper
      object_store: s3
      schema: v11
      index:
        prefix: loki_index_
        period: 24h

storage_config:
  boltdb_shipper:
    active_index_directory: /loki/index
    cache_location: /loki/index_cache
    shared_store: s3
  aws:
    bucketnames: ${AWS_S3_BUCKET}
    region: ${AWS_S3_REGION}
    access_key_id: ${AWS_S3_ACCESS_KEY_ID}
    secret_access_key: ${AWS_S3_SECRET_ACCESS_KEY}

compactor:
  working_directory: /loki/compactor
  shared_store: s3
  compaction_interval: 10m
  retention_enabled: true
  retention_delete_delay: 2h
  retention_delete_worker_count: 150

limits_config:
  # retain logs for 90 days
  retention_period: 2160h

and the data is still in s3?

Yes. The data is still in S3

maybe grafana uses some sort of guid to reference the S3 bucket instead of by name and the new container has now a new guid reference?

Can you spin down and spin up the S3 bucket and reconnect manually from the new docker instance to verify this hypothesis?

or open up the json of the dashboard and see if the guid keeps changing for the S2 reference

What you mean by spin down and spin up the S3 bucket? You mean deleting it and recreating it?

never mind, you can leave it as is. Can you check the following?

open up the json of the dashboard and see if the guid keeps changing for the S2 reference

Then recreate the grafana container again and see if the reference to S3 is somehow different?

I am actually not using Grafana at all. I just have Loki running and I am querying Loki directly. Do you know if there is there something similar I can check within Loki to verify the reference to S3?

1 Like

shared_store should probably be aws there.

Tried that without luck. According to the documentation seems like the allowed values for shared_store are: gcs, s3, azure, filesystem

Then I probably remember incorrectly. The thing you need to watch for is that the shared_store configured for boltdb_shipper needs to match with one storage configuration.

Take a look at the answer in this thread: amazon s3 - Loki config with s3 - Stack Overflow

Thanks for the link. I have tried all combinations. Nothing seems to be working. It’s quite weird that the documentation on this is so bad. I tried copy pasting the examples in the documentation as is, but they don’t seem to work

Let me share my configuration with you then:

common:
  storage:
    s3:
      bucketnames: <NAME>
      region: <REGION>
      endpoint: <VPC_ENDPOINT> # if you use VPC endpoint
      sse_encryption: true
      s3forcepathstyle: true

schema_config:
  configs:
  - from: 1990-01-01
    store: boltdb-shipper
    object_store: s3
    schema: v11
    index:
      prefix: index_
      period: 24h

storage_config:
  boltdb_shipper:
    active_index_directory: /loki/index
    cache_location: /loki/index_cache
    shared_store: s3