Pyroscope: Access Denied to S3 object storage

Grafana Pyroscope
1

I’m running Pyroscope in a docker container on an ec2 instance in AWS. I configured Pyroscope’s object storage for S3. When I start it up, it prints errors such as this, and then the process dies

ts=2024-08-23T22:58:31.695121133Z caller=compactor.go:585 level=error component=compactor component=compactor msg="failed to discover users from bucket" err="Access Denied"

...

ts=2024-08-23T22:58:32.70247757Z caller=module_service.go:118 level=warn msg="module failed with error" module=tenant-settings err="upload s3 object: Access Denied"
ts=2024-08-23T22:58:32.702555931Z caller=phlare.go:436 level=error msg="module failed" module=tenant-settings error="upload s3 object: Access Denied"

I am using an IAM instance profile to give the ec2 instance permission to access the bucket. I was able to install the aws cli on the ec2 instance and issue a list bucket on my bucket and have it succeed.

The IAM Policy is currently all encompassing for debugging purposes:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "PyroscopeStorage",
            "Effect": "Allow",
            "Action": [
                "s3:*"
            ],
            "Resource": [
                "arn:aws:s3:::my-bucket-name",
                "arn:aws:s3:::my-bucket-name/*"
            ]
        }
    ]
}

The configuration is:

multitenancy_enabled: true

server:
  log_level: debug

storage:
  backend: s3
  s3:
    bucket_name: my-bucket-name
    region: us-west-2
    endpoint: s3.us-west-2.amazonaws.com
    secret_access_key: ""
    access_key_id: ""
    sse:
      type: SSE-S3
    http:
      idle_conn_timeout: 1m30s
      response_header_timeout: 2m0s
      insecure_skip_verify: false
      tls_handshake_timeout: 10s
      expect_continue_timeout: 1s
      max_idle_connections: 100
      max_idle_connections_per_host: 100
      max_connections_per_host: 0

ingester: 
  lifecycler: 
    ring: 
      kvstore: 
        store: inmemory 

analytics:
  reporting_enabled: false

I have set up mimir and loki also with an s3 storage backend, and they work correctly (different buckets, but same ec2 instance and same approach).

An additional detail: S3 is being accessed via a VPC Gateway Endpoint.

What could the issue be? Thanks

2

Check your Cloudtrail logs and find reason why requests were rejected and if their source is expected iam role.

3

@jangaraj Thanks for the tip! I looked into CloudTrail and found that it is being reported as an “anonymous” user identity to S3, which explains why it isn’t working.

I also added a comment to this thread I found in github issues: Add support for standard AWS authentication in s3 backend. · Issue #2339 · grafana/pyroscope · GitHub

Is the IAM instance profile approach supposed to work with Pyroscope?

I will try creating an access key and secret as a workaround.