Create custom metrics from Tempo span attributes

Grafana Tempo
1

Hello. I have instrumented a Java application with OpenTelemetry using both auto and manual instrumentation. I am only instrumenting for traces, not metrics or logs. My traces are being sent to my OTel collector and then exported to Tempo. I see my traces in Tempo. I’m running everything from my local Docker.

In my spans, I have a span attribute named “database.cache.layer” and my goal is to create metrics from this span attribute. This metrics will be used in a dashboard to assist with building our SLI’s and SLO’s.

I’ve read this traceToMetrics intro which seems to fit my goal. In my Tempo config I’ve tried using the tags “service.name”, “service_name”, and "“database.cache.layer” to varying to degrees but when I click on the “Explore metrics for this span” button I just get “No data”. My assumption is that I’m using the wrong tag.

That leads me to several questions:

  1. Is it possible to create custom metrics from spans or am I limited to RED metrics?
  2. How do I verify which tags I can use when setting up Tempo Traces to Metrics?
  3. Is there a better approach to my goal?

For reference here are several configs that I’m using:
tempo.yaml

server:
  http_listen_port: 3200

distributor:
  receivers:
    jaeger:
      protocols:
        thrift_http:
        grpc:
        thrift_binary:
        thrift_compact:
    zipkin:
    otlp:
      protocols:
        http:
        grpc:
    opencensus:

ingester:
  max_block_duration: 5m

compactor:
  compaction:
    block_retention: 1h

metrics_generator:
  registry:
    external_labels:
      source: tempo
      cluster: docker-compose
  storage:
    path: /tmp/tempo/generator/wal
    remote_write:
      - url: http://prometheus:9090/api/v1/write
        send_exemplars: true

storage:
  trace:
    backend: local
    wal:
      path: /tmp/tempo/wal
    local:
      path: /tmp/tempo/blocks

overrides:
  metrics_generator_processors: [service-graphs, span-metrics]

prometheus.yaml

global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: "prometheus"
    scrape_interval: 5s
    static_configs:
      - targets: ["prometheus:9090"]

  - job_name: "prometheus-exporter-endpoint"
    scrape_interval: 5s
    static_configs:
      - targets: ["otel-col:8889"]

  - job_name: "otel-col-metrics"
    scrape_interval: 5s
    static_configs:
      - targets: ["otel-col:8888"]

  - job_name: "jaeger-metrics"
    scrape_interval: 5s
    static_configs:
      - targets: ["jaeger:14269"]

  - job_name: "tempo-metrics"
    scrape_interval: 5s
    static_configs:
      - targets: ["tempo:3200"]

  - job_name: "loki-metrics"
    scrape_interval: 5s
    static_configs:
      - targets: ["loki:3100"]

otel-col.yaml

receivers:
  otlp:
    protocols:
      grpc:
      http:
exporters:
  logging:
    verbosity: detailed
  loki:
    endpoint: http://loki:3100/loki/api/v1/push
    tls:
      insecure: true
  jaeger:
    endpoint: jaeger:14250
    tls:
      insecure: true
  otlp:
    endpoint: tempo:4317
    tls:
      insecure: true
processors:
  filter/ottl:
    error_mode: ignore
    traces:
      span:
        - 'name == "S3.ListObjects"'
        - 'attributes["rpc.method"] == "ListObject"'
extensions:
  health_check:
  pprof:
  zpages:
service:
  extensions: [health_check, pprof, zpages] 
  pipelines:
    traces:
      receivers: [otlp]
      processors: [filter/ottl]
      exporters: [logging, otlp]
    metrics:
      receivers: [otlp]
      exporters: [logging]
    logs:
      receivers: [otlp]
      exporters: [logging, loki]
1 Like
2

I think I found my answer here

Additional user defined labels can be created using the dimensions configuration option. When a configured dimension collides with one of the default labels (e.g. status_code), the label for the respective dimension is prefixed with double underscore (i.e. __status_code).

Custom labeling of dimensions is also supported using the dimension_mapping configuration option.

 # Additional dimensions to add to the metrics along with the intrinsic dimensions.
            # Dimensions are searched for in the resource and span attributes and are added to
            # the metrics if present.
            [dimensions: <list of string>]