Tempo not starting due to s3 storage permission issue

Grafana Tempo
1

while installing the tempo service we are getting below error, when checked all the permission is there for that role.

and checked there is no ListObjects action only for aws s3 role.

Attaching to tempo
tempo | level=info ts=2023-07-24T07:59:16.967853292Z caller=main.go:221 msg=“initialising OpenTracing tracer”
tempo | level=info ts=2023-07-24T07:59:16.974292532Z caller=main.go:108 msg=“Starting Tempo” version=“(version=r105-5132d08, branch=r105, revision=5132d0808)”
tempo | level=error ts=2023-07-24T07:59:18.003974575Z caller=main.go:111 msg=“error running Tempo” err=“failed to init module services error initialising module: store: failed to create store unexpected error from ListObjects on dev: Access Denied”
tempo exited with code 1

2

Hi @rajasudha225. You need to configure the following permissions in S3: Amazon S3 permissions | Grafana Tempo documentation. I hope that helps.

3

thank you for reply, i tried that also not worked. try below policy also but not working.

{
“Version”: “2012-10-17”,
“Id”: “Policy”,
“Statement”: [
{
“Sid”: “TempoPermissions”,
“Effect”: “Allow”,
“Principal”: {
“AWS”: “arn:aws:iam::xxxxxx:root”
},
“Action”: “s3:",
“Resource”: [
“arn:aws:s3:::dev”,
"arn:aws:s3:::dev/”
]
}
]
}

4

You need to add the following actions to your policy

            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:ListBucket",
                "s3:DeleteObject",
                "s3:GetObjectTagging",
                "s3:PutObjectTagging"
            ],

Also, are you correctly passing the authentication? There are a few methods supported.

5

in that policy i have added

“Action”: “s3:*”,

authentication method i am using ec2 role based with s3 full access.

6

can someone help me with the solution?

7

Can you verify that the resource is accessible from the EC2 instance? Not sure what’s failing. It seems that something’s not configured correctly.

8

below is the storage config i used and i am running as docker-compose and i checked for s3 we do have and working.

storage:
trace:
backend: s3
s3:
bucket: dev-api-alb-logs
endpoint: s3.ap-south-1.amazonaws.com
region: ap-south-1
insecure: true
tags: {‘test’: ‘test1’}
block:
bloom_filter_false_positive: .05
wal:
path: /tmp/tempo/wal
pool:
max_workers: 100
queue_depth: 10000

9

its resolved now. issue with i am policy.

thanks for the support.