Fetch logs from an S3 bucket to Loki?

Does anyone have experience with fetching logs from an S3 bucket to Loki?

My bucket is S3-compatible and is not an AWS service [^1], so I can’t use lambda-promtail to solve the problem.

Another idea that comes to my mind is to use Logstash with S3 input plugin for feeding the logs to Loki instead. But I don’t really want to provision an additional Logstash. Is there any way to solve the problem with Promtail?

This is a duplicated question. The thread is locked though, and I don’t particularly like the process of mounting an S3 bucket as a virtual filesystem.

[^1] For anyone curious, it’s this OBS (Object Storage Service) by a cloud provider in Germany.

1 Like

If you are set to using promtail then S3FS is probably your best bet. I haven’t used it personally but I’ve heard it’s reasonably stable.

Another solution that we do sometimes is to just use a simple script to pull from S3 bucket. Obviously you’ll need to do some coding here, and depending on how you structure your bucket (or don’t structure) how you approach the scripting would be different.

Have you figured out a better way than scripting? I have the same problem with Azure stotage account, and was thinking of using Logstash plugin with Azure as input.