Tempo distributed not showing tracing in grafana

[sbhushankr]

Hi, We redeployed everything and route the traffic with default url on load balancer then we able to get the tracing and able to open the same in grafana.

Now we are facing the issue to send the trace file on s3 bucket. Below error is coming in ingestor pod log.

level=error ts=2022-07-28T08:12:27.933472907Z caller=flush.go:221 org_id=single-tenant msg=“error performing op in flushQueue” op=1 block=58949f9e-ac2a-42d4-8734-106493dcc9af attempts=545 err=“error copying block from local to remote backend: error writing object to s3 backend, object single-tenant/58949f9e-ac2a-42d4-8734-106493dcc9af/data: Access Denied”

EKS worker EC2 server associated role has access to s3 bucket and we are passing the below parameter in config.

storage:
trace:
backend: s3
s3:
bucket: mulesoft-eks-tempo-logs-dev
endpoint: s3.dualstack.us-west-2.amazonaws.com
insecure: true

Regards,
Shashi

Regards,
Shashi

[annanayagarwal]

Hi @sbhushankr,

Tempo containers require the following permissions to enable writing/reading data from S3. Could you please confirm that the mentioned access policy has been applied?

S3 Permissions

[sbhushankr]

Hi @annanayagarwal , We go through the shared docs and we are using here the EC2 role access. so we have granted the s3 bucket access to role and that role is associated with EKS worker node. We checked the same to login on the worker node and able to push the file on s3 bucket.

Below policy associated with eks worker node.

Regards,
Shashi

[saritatewari4]

Hello @sbhushankr ,

did it work for you. I am facing the same issue. Could you please share the solution you followed.