Failed listing tenant blocks: error finding objects in s3 bucket, bucket: tempo: Truncated response should have continuation token set

Grafana Tempo
1

Configure the automatic expiration cleanup for Tempo. The compactor configuration in tempo.yml is as follows:

compactor:
  ring:
    kvstore:
      store: memberlist
    instance_interface_names:
      - eth0
    instance_addr: <IP>
    instance_port: 9095
  compaction:
    compaction_cycle: 60s
    compaction_window: 600s
    block_retention: 10h
    compacted_block_retention: 60m
    retention_concurrency: 10
    v2_in_buffer_bytes: 5242880
    v2_out_buffer_bytes: 20971520
    max_time_per_tenant: 20m

When the number of files in the bucket is small, cleanup succeeds. However, if the number increases slightly, the following error appears in the logs, preventing automatic cleanup.

error finding objects in s3 bucket, bucket: tempo: Truncated response should have continuation token set
error finding objects in s3 bucket, bucket: tempo: Truncated response should have continuation token set"

I suspect this occurs because AWS requests return a maximum of 1000 items by default. Beyond this limit, pagination is required, and Tempo fails to handle pagination properly, causing the error. Is my understanding correct? If so, how can this be resolved?

2

supplement the storage configuration

storage:
    trace:
        blocklist_poll: 20m0s
        blocklist_poll_concurrency: 50
        blocklist_poll_tenant_concurrency: 0
        blocklist_poll_fallback: true
        blocklist_poll_tenant_index_builders: 2
        blocklist_poll_stale_tenant_index: 0s
        blocklist_poll_jitter_ms: 0
        blocklist_poll_tolerate_consecutive_errors: 10
        blocklist_poll_tolerate_tenant_failures: 10
        empty_tenant_deletion_enabled: false
        empty_tenant_deletion_age: 0s
        backend: s3
        s3:
            bucket: tempo
            prefix: ""
            endpoint: '********'
            region: ""
            access_key: '********'
            secret_key: '********'
            session_token: ""
            insecure: true
            part_size: 0
            hedge_requests_at: 0s
            hedge_requests_up_to: 2
            signature_v2: true
            forcepathstyle: true
            enable_dual_stack: false
            bucket_lookup_type: 0
            tags: {}
            storage_class: ""
            metadata: {}
            native_aws_auth_enabled: false
            list_blocks_concurrency: 1

version ：

tempo, version 2.7.0 (branch: HEAD, revision: b0da6b481)
  build user:       
  build date:       
  go version:       go1.23.4
  platform:         linux/amd64
  tags:             unknown