TraceQL metrics local-blocks only holds last 30 min

Grafana Tempo
1

I have enabled Metrics Generator with local blocks only for out distributed tempo deployment to test it’s potential .
The problem I are facing now is that only the last 30m window of time is TraceQL metrics available to test . I have deployed Metrics Generator as a Stateful Set with persistence enabled.

I don’t want necessary to increase the window , I only want to run the query with an interval of 10m starting from 5h ago.
The rendered Metrics Generator config inside the k8s cluster is :

    metrics_generator:
      metrics_ingestion_time_range_slack: 30s
      processor:
        local_blocks:
          filter_server_spans: false
        service_graphs:
          dimensions: ]
          histogram_buckets:
          - 0.1
          - 0.2
          - 0.4
          - 0.8
          - 1.6
          - 3.2
          - 6.4
          - 12.8
          max_items: 10000
          wait: 10s
          workers: 10
        span_metrics:
          dimensions: ]
          histogram_buckets:
          - 0.002
          - 0.004
          - 0.008
          - 0.016
          - 0.032
          - 0.064
          - 0.128
          - 0.256
          - 0.512
          - 1.02
          - 2.05
          - 4.1
      registry:
        collection_interval: 15s
        external_labels: {}
        stale_duration: 15m
      ring:
        kvstore:
          store: memberlist
      storage:
        path: /var/tempo/wal
        remote_write:
        - metadata_config:
            send_interval: 1m
          send_exemplars: true
          tls_config:
            insecure_skip_verify: true
          url: http://prom-stack-kube-prometheus-prometheus.monitoring.svc.cluster.local:9090/api/v1/write
        remote_write_add_org_id_header: true
        remote_write_flush_deadline: 1m
        wal: null
      traces_storage:
        path: /var/tempo/traces
    multitenancy_enabled: false
    overrides:
      defaults:
        metrics_generator:
          processors:
          - local-blocks
      per_tenant_override_config: /runtime-config/overrides.yaml

Sample output for 1h

image
image1862×780 62.2 KB

Is there a way to increase the ‘retention’ of local blocks ?

2

Hi yes, to calculate metrics on older data we need to configure the generator/local-blocks to flush them to storage:

metrics_generator:
  processor:
    local_blocks:
      flush_to_storage: true

This flushes the special blocks to storage, and it uses the same retention as other data. Be aware this will increase storage costs because it is flushing additional blocks. We will eliminate this in the future with a new architecture, but for now it is required. There is some additional info in the 2.6 release notes.

1 Like