Loki with multiple aws accounts

[dougbyrne]

Hi, I’m interested in using Loki as a way to aggregate logs from multiple AWS accounts.

Ideally, I’d like to have the write path running within each individual account, and writing to a S3 bucket within that account. Then, have a single read path that can read from multiple buckets. Having the data for each account stay within that account makes cost attribution easier, and prevents one account from taking down the ingestion for all accounts.

I’ve been looking at the documentation, and I’ve not figured out a couple things:

• Is it possible to have multiple buckets in use at once, e.g. one per tennant?
• Can the AWS authentication use assume-role credentials, to cross the account boundaries?
• The Queriers will not be able to contact the Ingesters, is that required?

Am I thinking about this the wrong way? Do I instead need to have a Read Path in each AWS account, with a single Grafana configured to read from each Loki stack? Or should I centralize the Write path?

[tonyswumac]

@dougbyrne

• Is it possible to have multiple buckets in use at once, e.g. one per tennant?

I don’t think this is possible.

@dougbyrne

• Can the AWS authentication use assume-role credentials, to cross the account boundaries?

Not related to Loki directly, but yes.

@dougbyrne

• The Queriers will not be able to contact the Ingesters, is that required?

Required if you want to be able to query logs that aren’t written to chunk storage yet.

Unless there is a good reason to separate your logs, I’d recommend aggregating logs into one Loki cluster running in your centralized account. Otherwise, you might as well invest in your automated deployment procedure and deploy one Loki per account.