How move chunks and index files from s3 to local filesystem?

Grafana Loki
2

Thanks @evgenyluvsandugar for posting this here, the question gets asked enough that hopefully it will help others to see the answer here:

Yes this should be possible (and the reverse possible too)*

* I’ve never tried this

The chunks themselves can be copied from s3 to filesystem or vice versa, their content wouldn’t change, what does need to change is the names and the location.

In an object store like S3 (or GCS or Azure Blob) the directory structure looks like this:

bucket_name
    /index
    /tenantID1
    /tenantID2
   ...

If you are using Loki in single user mode, you would see:

bucket_name
    /index
    /fake

As Loki choses the tenant ID of fake (sorry this isn’t the best name but it’s almost impossible to change at this point)

On the local filesystem store the layout is a little different,

/path/to/filesystem/store
  /index
  /ZmFrZS85MTg1NDQzM2Q0YzlkNjlmOjE3NGZlMGYyMDE1OjE3NGZlNDdmODgzOjU4NWMxMTU3
  /ZmFrZS85MTg1NDQzM2Q0YzlkNjlmOjE3NjcyNmI1ODYxOjE3NjcyN2Q1NTBkOjc5NGQyMThh
  /ZmFrZS85MTg1NDQzM2Q0YzlkNjlmOjE3NGZlNGZjMDJjOjE3NGZlOTUwZDUzOjk1OGY5ZTU5    
  /ZmFrZS85MTg1NDQzM2Q0YzlkNjlmOjE3NjcyY2JiNmNlOjE3NjcyZWMwOWFmOmNlMWQ3ZTdh
  ...

The path here is what’s defined in the config like so

storage_config:
  filesystem:
    directory: /path/to/filesystem/store

those really long names are actually base 64 encoded names and if you decode one you find:

fake/91854433d4c9d69f:174fe0f2015:174fe47f883:585c1157

So, this becomes the problem, converting between an object store and the filesystem store requires appending the tenantID to the chunk name and then base64 encoding it and storing that in the filesystem directory.

The index folder can be copied as is I believe (and is only present if you are using boltdb-shipper index type)

Unfortunately nobody has built any tooling to do this yet as far as I know.

I’m sorry this isn’t probably the answer you were hoping for but it should be possible to do this although it would very likely require building a tool to accomplish.

6 Likes