Problem to add multiple dimensions on span metrics tempo

Grafana Tempo
1

Hi everyone,

I’m having an issue with Tempo and Prometheus when configuring span_metrics. I’m trying to include two dimensions like this:

span_metrics:
dimensions:

  • browserId
  • report.type
    filter_policies:
  • include:
    attributes:
    • key: resource.service.name
      value: ms-browser
    • key: name
      value: GetData
      match_type: strict
  • include:
    attributes:
    • key: resource.service.name
      value: ms-reports
    • key: name
      value: GetReport
      match_type: strict
      histogram_buckets:
  • 0.002
  • 0.004
  • 0.008
  • 0.016
  • 0.032
  • 0.064
  • 0.128
  • 0.256
  • 0.512
  • 1.02
  • 2.05
  • 4.1

Notes:

  • If I remove browserId and leave only report.type, the metric appears, but the dimension doesn’t show up.
  • If I include both browserId and report.type, nothing works — no metrics appear at all.
  • Has anyone faced this issue or has a working example of span_metrics using multiple dimensions?

Thanks a lot in advance for your help!

2

check your logs, and you will have more clarity on why it’s failing.

remote writes might be failing due to browserId containing utf-8 or other invalid label values for Prometheus remote write.

3

Hello, thank you! I didn’t see anything wrong in the logs, but we managed to fix the issue, and the configuration is now working on my side :slightly_smiling_face:

yaml

metrics_generator:
  processor:
    span_metrics:
      dimensions:
      - browserId
      - "report.type"
      filter_policies:
      - include:
          match_type: regex
          attributes:
          - key: resource.service.name
            value: ms-browser|ms-reports
          - key: kind
            value: SPAN_KIND_SERVER|SPAN_KIND_INTERNAL
      histogram_buckets:
      - 0.002
      - 0.004
      - 0.008
      - 0.016
      - 0.032
      - 0.064
      - 0.128
      - 0.256
      - 0.512
      - 1.02
      - 2.05
      - 4.1

It seems that expressions with dots (.) or certain characters might not be fully supported by the metrics_generator in Tempo. I’m currently using version 2.4.

1 Like