S3 image upload for alerts

Hi
I have configured in grapha.ini the external image storage for s3, but when the alert gets sent I get access denied. I have checked my IAM and it def works so its not an AWS IAM issue.

Could there be an issue with having a forward slash in the secret key maybe? Can I use quotes around the key and secret key for s3 credentials.

This is my error message in the log.

t=2019-05-08T15:01:47+0100 lvl=eror msg=“Failed to upload alert panel image.” logger=alerting.notifier error=“AccessDenied: Access Denied\n\tstatus code: 403, request id: E0C5E4BC0AC0EDD8, host id: aB+OgVS+n1op4ijb5N1cjSn4R3eCVere/gdxrC1iN+rqnYzVTudDetE9IMn1NzYh6jSoHKAdMj4=”

It really looks like an IAM error. Are you sure have granted the correct permissions? From the docs:

Access key requires permissions to the S3 bucket for the ‘s3:PutObject’ and ‘s3:PutObjectAcl’ actions.

From googling this error:

Hi Daniel
Thanks for your response. Yes i have granted permissions correctly, as I tested uploading an object to the bucket using the aws cli tools and I can create and delete objects using the access and secret key used.

This is why I am totally confused. Only thing I can think of is the secret has a backslash in it, could that be causing a problem?

It could in theory. I just did a quick test at the code level and it seems like the code that reads the ini file can handle a backslash. But maybe it needs to be escaped before sending to the S3 api. Could you give me an example of a secret key with the characters you think might be the problem?

But I would really double check that you have set the PutObjectAcl action correctly in IAM.

Thanks Daniel

The IAM is defo setup correctly otherwise I would not be able to upload an object via the CLI tool
This is the policy that I assigned to my user

{
“Statement”: [
{
“Effect”: “Allow”,
“Action”: [
“s3:ListBucket”,
“s3:GetBucketLocation”,
“s3:ListBucketMultipartUploads”
],
“Resource”: “arn:aws:s3:::bucketname”,
“Condition”: {}
},
{
“Effect”: “Allow”,
“Action”: [
“s3:AbortMultipartUpload”,
“s3:DeleteObject”,
“s3:DeleteObjectVersion”,
“s3:GetObject”,
“s3:GetObjectAcl”,
“s3:GetObjectVersion”,
“s3:GetObjectVersionAcl”,
“s3:PutObject”,
“s3:PutObjectAcl”,
“s3:PutObjectAclVersion”
],
“Resource”: “arn:aws:s3:::bucketname/",
“Condition”: {}
},
{
“Effect”: “Allow”,
“Action”: “s3:ListAllMyBuckets”,
“Resource”: "
”,
“Condition”: {}
}
]
}

The secret key looks like follows (the x’s are standard alphanumeric characters)

secret_key = xxxxxx+xxxxxxxYQ1/VM

I am also facing the same issue, using the latest Grafana Image - v6.7.3, and build custom image with grafama-image-renderer. My AWS keys do not have any special characters, I tried giving S3 Full Access, even then, I get AccessDenied: Access Denied\n\tstatus code: 403

Hey, I faced the same issue - just make sure you have “Block public access (bucket settings)” disabled in the S3 bucket settings (AWS blocks any attempts to make objects public by default)

The issues is not in “Block Public Access” settings. That is for the client to view any uploaded image.
Since the logs are from grafana container and in the upload image request, somehow grafana was not able to authenticate to S3 using the keys provided.
I resolved my issues by changing some settings of grafana-image-renderer (dont remember exactly what settings)

Hello , I am getting the same issue . How did you resolve it ? Can you share me the knowledge?

Hello!
The same issue:( With aws cli it works, but in Grafana it doesn’t work. How to fix it?

I found the problem.

  • It needs to enable ACL on bucket.
  • Set up the following “Block public access” settings like the following on the screenshot: