How to migrate the data from filesystem to gcs

Grafana Loki
,
1

I recently moved my loki storage to gcs bucket, earlier it was configured and used pvc, now I tried to move the chunks and index but the index file name it look different for gcs and it is in compressed format, for example compactor-1715253589.gz. Can anyone help me with this?

2

I think it’s possible to directly copy index and chunk files from filesystem to object storage, but I haven’t tried, so I can’t tell you precisely how to do it. The format could also be different because you are using different index configuration, not sure. If you are attempting to do this. I’d say:

  1. Make sure the index format matches.
  2. You should have at least 2 index configurations, one for your index and chink files copied from old cluster, one for anything new.

If that doesn’t work, then I’d recommend you to instead consider other workarounds:

  1. Use Loki api, export from existing cluster then import into new cluster.
  2. Leave the existing cluster as is. Then once retention is up then you can get rid of old cluster. You can use multiple data sources in Grafana in case people want to query old cluster.
3

Hey @tonyswumac thanks for the response, I have tried to copy and directly store the index and chunks in gcp but the application is failed due to that. and I am not sure what should I change in the configuration part for storage, it looks same for both filesystem and gcs, but it storing the datas in different format.
here is the config,

compactor:
  shared_store: filesystem
  working_directory: /data/loki/boltdb-shipper-compactor
ingester:
  chunk_block_size: 262144
  chunk_idle_period: 3m
  chunk_retain_period: 1m
  lifecycler:
    ring:
      replication_factor: 1
  max_transfer_retries: 0
  wal:
    dir: /data/loki/wal
limits_config:
  enforce_metric_name: false
  max_entries_limit_per_query: 5000
  reject_old_samples: true
  reject_old_samples_max_age: 168h
memberlist:
  join_members:
  - 'loki-memberlist'
schema_config:
  configs:
  - from: "2020-10-24"
    index:
      period: 24h
      prefix: index_
    object_store: filesystem
    schema: v11
    store: boltdb-shipper
server:
  grpc_listen_port: 9095
  http_listen_port: 3100
storage_config:
  boltdb_shipper:
    active_index_directory: /data/loki/boltdb-shipper-active
    cache_location: /data/loki/boltdb-shipper-cache

  config:
    auth_enabled: false
    chunk_store_config:
      max_look_back_period: 0s
    compactor:
      shared_store: gcs
      working_directory: /data/loki/boltdb-shipper-compactor
    ingester:
      chunk_block_size: 262144
      chunk_idle_period: 3m
      chunk_retain_period: 1m
      lifecycler:
        ring:
          replication_factor: 1
      max_transfer_retries: 0
      wal:
        dir: /data/loki/wal
    limits_config:
      enforce_metric_name: false
      max_entries_limit_per_query: 500000000
      reject_old_samples: true
      reject_old_samples_max_age: 168h
    memberlist:
      join_members:
      - 'loki-memberlist'
    schema_config:
      configs:
      - from: "2020-10-24"
        index:
          period: 24h
          prefix: index_
        object_store: gcs
        schema: v11
        store: boltdb-shipper
    server:
      grpc_listen_port: 9095
      http_listen_port: 3100
    storage_config:
      boltdb_shipper:
        active_index_directory: /data/loki/boltdb-shipper-active
        cache_location: /data/loki/boltdb-shipper-cache
        cache_ttl: 24h
        shared_store: gcs
      filesystem:
        directory: /data/loki/chunks
      gcs:
        bucket_name: supra-loki-storage
      boltdb_shipper:
        active_index_directory: /data/loki/boltdb-shipper-active
        cache_location: /data/loki/boltdb-shipper-cache
        cache_ttl: 24h
        shared_store: gcs
4
  1. Did you make sure the directory structure remains identical?
  2. What are the errors?