Viewing logs that have already been moved to s3 glacier

Grafana Loki
1

The current configuration of Loki in my environment has S3 as storage.
The “index” are separated into 24-hour blocks, and the “chuncks” into random blocks.
There is a policy that moves files older than 30 days to the “glacier” storage class.
I need to view logs older than 30 days.
I have already restored the corresponding index and the “chuncks” to the S3 default class as well, but I cannot view the logs.

My config:

  schema_config:
    configs:
    - from: "2022-01-11"
      index:
        period: 24h
        prefix: loki_index_
      store: boltdb-shipper
      object_store: s3
      schema: v12

chuncks
index

Error:

error
error1802×265 13.6 KB

2

Error says InvalidObjectState, how long ago did you do the restore from Glacier back to S3? It takes quite a while if I remember correctly.

3

Ok, in the S3 bucket I can already download the content so I assume that the content has already been restored to the default class. The restoration was requested yesterday, and usually takes about 4 hours at most.

4

Another possibility is you have other chunk files that aren’t restored. Check logs and see if it gives you the path, or you can check cloudtrail and see if it gives you the path, and work from there.

But this is not a Loki issue.

5

I made a script in Python to find out the chunks that I should restore, for this I consider the date of the last modification. And I have already restored them all but the error persists.

Arquivo: fake/100507c352b27a42/18f83ee324d:18f83ee6d0a:49ebfb8e, Modificado em: 2024-05-17 00:53:14+00:00
Arquivo: fake/100507c352b27a42/18f85c7938a:18f85c7938b:88caf601, Modificado em: 2024-05-17 09:30:14+00:00
Arquivo: fake/100507c352b27a42/18f85fe8261:18f85fe8262:c4bbe31f, Modificado em: 2024-05-17 10:30:14+00:00
Arquivo: fake/100507c352b27a42/18f86699b51:18f86e7e1d9:94043f2, Modificado em: 2024-05-17 14:15:14+00:00
Arquivo: fake/100507c352b27a42/18f86eb55e7:18f87604f8a:9a528fdf, Modificado em: 2024-05-17 16:26:44+00:00
Arquivo: fake/100507c352b27a42/18f87790adc:18f87e71478:ffa220c0, Modificado em: 2024-05-17 18:53:44+00:00
Arquivo: fake/100507c352b27a42/18f87e8896d:18f882f94e3:5ae57baf, Modificado em: 2024-05-17 20:43:14+00:00
Arquivo: fake/100507c352b27a42/18f88543f24:18f8883383e:3f475dd6, Modificado em: 2024-05-17 22:14:14+00:00
Arquivo: fake/1005a9f9fd2b5d7a/18f885a15cb:18f88607c3c:d1ab7d43, Modificado em: 2024-05-17 21:36:23+00:00
Arquivo: fake/1006f3e9a99dcfb6/18f8758c364:18f877ec381:7980933e, Modificado em: 2024-05-17 16:59:53+00:00
Arquivo: fake/1006f3e9a99dcfb6/18f877ec380:18f87a545f5:c39ab1c3, Modificado em: 2024-05-17 17:41:53+00:00
Arquivo: fake/1006f3e9a99dcfb6/18f87a545f4:18f87cb77b4:9a34ab04, Modificado em: 2024-05-17 18:23:53+00:00
Arquivo: fake/1006f3e9a99dcfb6/18f87cb77b3:18f87f40b03:bb4c8517, Modificado em: 2024-05-17 19:07:53+00:00
Arquivo: fake/1006f3e9a99dcfb6/18f87f40cf7:18f881a28bb:7d277640, Modificado em: 2024-05-17 19:49:53+00:00
Arquivo: fake/1006f3e9a99dcfb6/18f881a28ba:18f8843685f:776df8c7, Modificado em: 2024-05-17 20:34:53+00:00
Arquivo: fake/1006f3e9a99dcfb6/18f8843685e:18f886cac1b:7cf7f3d6, Modificado em: 2024-05-17 21:19:53+00:00
Arquivo: fake/1006f3e9a99dcfb6/18f886cac1a:18f889eeb45:eaf4bbe, Modificado em: 2024-05-17 22:14:53+00:00
Arquivo: fake/1006f3e9a99dcfb6/18f889eeb44:18f88b591b9:4735763a, Modificado em: 2024-05-17 23:09:23+00:00
6

Were you able to find the S3 file that Loki is trying to access with status error?

7

Yes, I can easily find them.
This should work, right? recover the “index” and “chucks” of glacier and be able to access the contents of the logs normally in grafana.

8

I’ve never personally done this before, but if you restore the object from glacier back to S3 I don’t see why it shouldn’t work.