Hello. I have instrumented a Java application with OpenTelemetry using both auto and manual instrumentation. I am only instrumenting for traces, not metrics or logs. My traces are being sent to my OTel collector and then exported to Tempo. I see my traces in Tempo. I’m running everything from my local Docker.
In my spans, I have a span attribute named “database.cache.layer” and my goal is to create metrics from this span attribute. This metrics will be used in a dashboard to assist with building our SLI’s and SLO’s.
I’ve read this traceToMetrics intro which seems to fit my goal. In my Tempo config I’ve tried using the tags “service.name”, “service_name”, and "“database.cache.layer” to varying to degrees but when I click on the “Explore metrics for this span” button I just get “No data”. My assumption is that I’m using the wrong tag.
That leads me to several questions:
- Is it possible to create custom metrics from spans or am I limited to RED metrics?
- How do I verify which tags I can use when setting up Tempo Traces to Metrics?
- Is there a better approach to my goal?
For reference here are several configs that I’m using:
server: http_listen_port: 3200 distributor: receivers: jaeger: protocols: thrift_http: grpc: thrift_binary: thrift_compact: zipkin: otlp: protocols: http: grpc: opencensus: ingester: max_block_duration: 5m compactor: compaction: block_retention: 1h metrics_generator: registry: external_labels: source: tempo cluster: docker-compose storage: path: /tmp/tempo/generator/wal remote_write: - url: http://prometheus:9090/api/v1/write send_exemplars: true storage: trace: backend: local wal: path: /tmp/tempo/wal local: path: /tmp/tempo/blocks overrides: metrics_generator_processors: [service-graphs, span-metrics]
global: scrape_interval: 15s evaluation_interval: 15s scrape_configs: - job_name: "prometheus" scrape_interval: 5s static_configs: - targets: ["prometheus:9090"] - job_name: "prometheus-exporter-endpoint" scrape_interval: 5s static_configs: - targets: ["otel-col:8889"] - job_name: "otel-col-metrics" scrape_interval: 5s static_configs: - targets: ["otel-col:8888"] - job_name: "jaeger-metrics" scrape_interval: 5s static_configs: - targets: ["jaeger:14269"] - job_name: "tempo-metrics" scrape_interval: 5s static_configs: - targets: ["tempo:3200"] - job_name: "loki-metrics" scrape_interval: 5s static_configs: - targets: ["loki:3100"]
receivers: otlp: protocols: grpc: http: exporters: logging: verbosity: detailed loki: endpoint: http://loki:3100/loki/api/v1/push tls: insecure: true jaeger: endpoint: jaeger:14250 tls: insecure: true otlp: endpoint: tempo:4317 tls: insecure: true processors: filter/ottl: error_mode: ignore traces: span: - 'name == "S3.ListObjects"' - 'attributes["rpc.method"] == "ListObject"' extensions: health_check: pprof: zpages: service: extensions: [health_check, pprof, zpages] pipelines: traces: receivers: [otlp] processors: [filter/ottl] exporters: [logging, otlp] metrics: receivers: [otlp] exporters: [logging] logs: receivers: [otlp] exporters: [logging, loki]