Hi Team,
We are getting raw logs from different sources into s3 buckets. We want the logs from these s3 buckets to be ingested into Loki. We are using Alloy to collect logs from sources like azure event hub, log files. Does Alloy provide the capability to scrape logs from S3 bucket. I see otelcol.exporter.awss3 but it is in experimenal stage. Could you please help understand the most reliable way to scrape logs from s3 and ingest into Loki?
Thanks!
That’s just simple HTTP request with specific payload/protocol (for example standard OTLP protocol). So find client/tool, which is able to do that (nobody is stopping you to also write own client in your favorite programming language and execute it in your favorite way) and it’s able read also from S3 (read = receive, so it must be otel receiver, not otel exporter if you want to use something from OTEL world).
Talk with your AWS architect if you want to do that in AWS native way. Usually S3 events are used, so they are often paired with Lambda for that execution part. But there is many options (e.g. AWS services which can be used for scalability, reliability).