Loki Write Pod Failing to Flush on an AWS S3 bucket with kms encryption

Grafana Loki
1

Hello i’m having issue with the loki write pod, i have several hours review all the policies and so, and i had not any luck at all, this is the message coming from loki write pod:

msg="failed to flush" err="failed to flush chunks: store put chunk: AccessDenied: User: arn:aws:sts::0XXXXXXXXX:assumed-role/testing-loki/1740857650077647399 is not authorized to perform: kms:GenerateDataKey on this resource because the resource does not exist in this Region, no resource-based policies allow access, or a resource-based policy explicitly denies access\n\tstatus code: 403

I’m using federated identities to provide access to loki , this is the policy it’s being assume by the federated identity.

{
    "Statement": [
        {
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:DeleteObject"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:s3:::testing-loki20240727202733654200000001",
                "arn:aws:s3:::testing-loki20240727202733654200000001/*"
            ]
        },
        {
            "Action": "s3:ListBucket",
            "Effect": "Allow",
            "Resource": [
                "arn:aws:s3:::testing-loki20240727202733654200000001",
                "arn:aws:s3:::testing-loki20240727202733654200000001/*"
            ]
        },
        {
            "Action": [
                "s3:ListBucket",
                "s3:PutObject",
                "s3:GetObject",
                "s3:DeleteObject"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:s3:::testing-loki20240727202733654200000001",
                "arn:aws:s3:::testing-loki20240727202733654200000001/*"
            ]
        },
        {
            "Action": [
                "kms:GenerateDataKey",
                "kms:Decrypt",
                "kms:Encrypt",
                "kms:ReEncryptTo",
                "kms:GenerateDataKeyWithoutPlaintext",
                "kms:DescribeKey",
                "kms:GenerateDataKeyPairWithoutPlaintext",
                "kms:GenerateDataKeyPair",
                "kms:ReEncryptFrom"
            ],
            "Effect": "Allow",
            "Resource": "arn:aws:kms:us-east-1:XXXXXXXXXXX:key/XXXXXXXX-8a5c-49ae-b1ef-XXXXXXXX"
        }
    ],
    "Version": "2012-10-17"
}

everything is under the same region, so not sure why cannot find it, not sure if is a missing action

2

Maybe you need to configure KMS policy. Maybe it’s rejected there - on the KMS resource policy level.

3

yep i checked that also, this is what i have

{
  "Version": "2012-10-17",
  "Id": "key-default-1",
  "Statement": [
    {
      "Sid": "Enable IAM User Permissions",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::XXXXXXXXXX:root"
      },
      "Action": "kms:*",
      "Resource": "*"
    }
  ]
}

it looks good, everything is created with terraform with only one provider and is set to a fixed region, and i double check the region was properly set, and was the same for all of the resources too

4

The issue is solved now, just for the sake of the reference, it wasn’t related to any loki configuration issue, we are using fluxcd to deploy all the system tools and flux was rolling it back to a previous failing version, with a typo in the s3 bucket :slight_smile:

thanks a lot for your help