# Tempo Metrics Generator not writing to prometheus

**URL:** <https://community.grafana.com/t/tempo-metrics-generator-not-writing-to-prometheus/109895>\
**Category:** Grafana Tempo\
**Created:** [December 13, 2023, 3:08am UTC](https://community.grafana.com/t/tempo-metrics-generator-not-writing-to-prometheus/109895 "2023-12-13T03:08:46Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![samuelreay](https://sea2.discourse-cdn.com/grafana/user_avatar/community.grafana.com/samuelreay/32/40512_2.png) [@samuelreay](https://community.grafana.com/u/samuelreay)\
**Post date:** [December 13, 2023, 3:08am UTC](https://community.grafana.com/t/tempo-metrics-generator-not-writing-to-prometheus/109895/1 "2023-12-13T03:08:46Z")

</div>

I’ve seen a fair few people trying to get this working, and it seems like the docs are missing an example for people to work from, so hopefully that can be the example.

Example prior post: [How enable Service Graph with Tempo](https://community.grafana.com/t/how-enable-service-graph-with-tempo/87715)

I’m facing the same issue as others, where I believe everything is configured and enabled, but there are no span metrics being written into prometheus, but no idea how to debug it and get it working.

Here’s how everything is set up right now:

`docker-compose.yml`

```yaml
version: "3.8"

services:
  loki:
    image: grafana/loki:2.9.3
    command: -config.file=/etc/loki/local-config.yaml
    ports:
      - "3100:3100"

  prometheus:
    image: prom/prometheus:v2.45.0
    ports:
      - "9090:9090"
    volumes:
      - ./etc/prometheus:/workspace
    command:
      - --config.file=/workspace/prometheus.yml
      - --enable-feature=remote-write-receiver
      - --enable-feature=exemplar-storage
    depends_on:
      - loki

  tempo:
    image: grafana/tempo:2.3.1
    command: 
      - "--target=all"
      - "--storage.trace.backend=local"
      - "--storage.trace.local.path=/var/tempo"
      - "--auth.enabled=false"
      - "--config.file=/etc/tempo/tempo.yaml"
    ports:
      - "14250:14250"
    depends_on:
      - loki
    volumes:
      - ./etc/tempo:/etc/tempo

  grafana:
    image: grafana/grafana:10.2.2
    ports:
      - "3000:3000"
    environment:
      - GF_AUTH_ANONYMOUS_ENABLED=true
      - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
      - GF_AUTH_DISABLE_LOGIN_FORM=true
    volumes:
      - ./etc/grafana/:/etc/grafana/provisioning/datasources
      - ./etc/dashboards.yaml:/etc/grafana/provisioning/dashboards/dashboards.yaml
      - ./etc/dashboards:/etc/grafana/dashboards
    depends_on:
      - loki
      - prometheus

```

Traces in other services (which I’ve removed from the compose for simplicity) are being received and show up fine in the trace explorer in tempo, but the service graph has its normal “This isnt configured” and going to prometheus and looking for the span metrics shows nothing is published.

My `tempo.yaml` is:

```yaml
metrics_generator:
  registry:
    external_labels:
      source: tempo
      cluster: docker-compose
  storage:
    path: /tmp/tempo/generator/wal
    remote_write:
    - url: http://prometheus:9090/api/v1/write
      send_exemplars: true

overrides:
  defaults:
    metrics_generator:
      processors: [service-graphs, span-metrics]

```

And my grafana `datasource.yaml` is:

```yaml
# config file version
apiVersion: 1

# list of datasources that should be deleted from the database
deleteDatasources:
  - name: Prometheus
    orgId: 1

# list of datasources to insert/update depending
# what's available in the database
datasources:
  # <string, required> name of the datasource. Required
  - uid: prometheus
    orgId: 1
    name: Prometheus
    type: prometheus
    typeName: Prometheus
    access: proxy
    url: http://prometheus:9090
    password: ''
    user: ''
    database: ''
    basicAuth: false
    isDefault: true
    jsonData:
      exemplarTraceIdDestinations:
        - datasourceUid: tempo
          name: TraceID
      httpMethod: POST
    readOnly: false
    editable: true
  - uid: tempo
    orgId: 1
    name: Tempo
    type: tempo
    typeName: Tempo
    access: proxy
    url: http://tempo
    password: ''
    user: ''
    database: ''
    basicAuth: false
    isDefault: false
    jsonData:
      serviceMap:
        datasourceUid: 'prometheus'
      nodeGraph:
        enabled: true
      search:
        hide: false
      lokiSearch:
        datasourceUid: loki
      tracesToLogs:
        datasourceUid: loki
        filterBySpanID: false
        filterByTraceID: true
        mapTagNamesEnabled: false
        tags:
          - service
    readOnly: false
    editable: true
  - uid: loki
    orgId: 1
    name: Loki
    type: loki
    typeName: Loki
    access: proxy
    url: http://loki:3100
    password: ''
    user: ''
    database: ''
    basicAuth: false
    isDefault: false
    jsonData:
      derivedFields:
        - datasourceUid: tempo
          # enableNameMatcher: true # swap when released: https://github.com/grafana/grafana/pull/76162
          matcherRegex: 'TraceID.*: "(\w+)"'
          name: trace_id
          url: $${__value.raw}
    readOnly: false
    editable: true

```

Traces coming through fine:

 ![image](https://us1.discourse-cdn.com/grafana/original/3X/0/b/0bf42ea05b4248c1ef98f4a758d27df97be2fe27.png)

Service graph isnt:

 ![image](https://us1.discourse-cdn.com/grafana/original/3X/e/f/ef2e42175def5f24a6d64be57a20f0bc855c1b98.png)

And the expected metrics in prometheus don’t exist:

 ![image](https://us1.discourse-cdn.com/grafana/original/3X/b/8/b88a32b251002bccd7d22707e536db53ffa8de19.png)

I’m unsure what else to try. The tempo.yaml seems to explicitly turn on the service-graphs and span-metrics processors in the default setting, and prometheus has its remote write feature enabled. Does anyone have any idea what could be going wrong?

---

<div class="post-metadata">

**Author:** ![vembry](https://avatars.discourse-cdn.com/v4/letter/v/7ba0ec/32.png) [@vembry](https://community.grafana.com/u/vembry)\
**Post date:** [January 26, 2024, 7:47pm UTC](https://community.grafana.com/t/tempo-metrics-generator-not-writing-to-prometheus/109895/2 "2024-01-26T19:47:01Z")

</div>

bump!

---

<div class="post-metadata">

**Author:** ![cbos](https://sea2.discourse-cdn.com/grafana/user_avatar/community.grafana.com/cbos/32/38315_2.png) [@cbos](https://community.grafana.com/u/cbos)\
**Post date:** [January 30, 2024, 9:17pm UTC](https://community.grafana.com/t/tempo-metrics-generator-not-writing-to-prometheus/109895/3 "2024-01-30T21:17:30Z")

</div>

@samuelreay @vembry

Note the feature you enabled on Prometheus for receiving remote writes is deprecated.  
You can use:

```
  - '--web.enable-remote-write-receiver'

```

To provide a setup with Tempo span metrics enabled, I created a sample setup:

> **[GitHub - cbos/observability-toolkit: Observability stack for local...](https://github.com/cbos/observability-toolkit)**
>
> Observability stack for local development and troubleshooting toolkit - GitHub - cbos/observability-toolkit: Observability stack for local development and troubleshooting toolkit

This is the Tempo config I use which is generating the span metrics correctly:  
[https://github.com/cbos/observability-toolkit/blob/main/config/tempo/tempo-config.yaml](https://github.com/cbos/observability-toolkit/blob/main/config/tempo/tempo-config.yaml)

I think you miss this under the `metrics_generator` config

```yaml

  processor:
    service_graphs:
    span_metrics:

```

---

<div class="post-metadata">

**Author:** ![vembry](https://avatars.discourse-cdn.com/v4/letter/v/7ba0ec/32.png) [@vembry](https://community.grafana.com/u/vembry)\
**Post date:** [January 31, 2024, 10:34am UTC](https://community.grafana.com/t/tempo-metrics-generator-not-writing-to-prometheus/109895/4 "2024-01-31T10:34:09Z")

</div>

> Note the feature you enabled on Prometheus for receiving remote writes is deprecated.  
> Y⁄ou can use:
> 
> ```auto
> - '--web.enable-remote-write-receiver'
> 
> ```

I added this to my prometheus command on docker compose, it accepts remote write now!

> I think you miss this under the `metrics_generator` config
> 
> ```auto
> processor:
> service_graphs:
> span_metrics:
> 
> ```

thanks! i ended up browsing [grafana/tempo’s docs](https://grafana.com/docs/tempo/latest/configuration/#metrics-generator) and found these configs you mentioned

---

<div class="post-metadata">

**Author:** ![shashankjb3](https://avatars.discourse-cdn.com/v4/letter/s/49beb7/32.png) [@shashankjb3](https://community.grafana.com/u/shashankjb3)\
**Post date:** [October 25, 2024, 9:57am UTC](https://community.grafana.com/t/tempo-metrics-generator-not-writing-to-prometheus/109895/5 "2024-10-25T09:57:04Z")

</div>

Hii can Tempo metrics-generator remote write metrics directly to mimir ? is it possible?

---

<div class="post-metadata">

**Author:** ![shashankjb3](https://avatars.discourse-cdn.com/v4/letter/s/49beb7/32.png) [@shashankjb3](https://community.grafana.com/u/shashankjb3)\
**Post date:** [October 25, 2024, 10:11am UTC](https://community.grafana.com/t/tempo-metrics-generator-not-writing-to-prometheus/109895/6 "2024-10-25T10:11:07Z")

</div>

![Screenshot 2024-10-25 142656](https://us1.discourse-cdn.com/grafana/original/3X/1/c/1c88f56393373e0a310e026516d3fc9335ac18cc.png)

---

<div class="post-metadata">

**Author:** ![shashankjb3](https://avatars.discourse-cdn.com/v4/letter/s/49beb7/32.png) [@shashankjb3](https://community.grafana.com/u/shashankjb3)\
**Post date:** [October 28, 2024, 7:01am UTC](https://community.grafana.com/t/tempo-metrics-generator-not-writing-to-prometheus/109895/7 "2024-10-28T07:01:09Z")

</div>

hii @cbos can you pls help me here ?

---

<div class="post-metadata">

**Author:** ![cbos](https://sea2.discourse-cdn.com/grafana/user_avatar/community.grafana.com/cbos/32/38315_2.png) [@cbos](https://community.grafana.com/u/cbos)\
**Post date:** [November 9, 2024, 2:47pm UTC](https://community.grafana.com/t/tempo-metrics-generator-not-writing-to-prometheus/109895/9 "2024-11-09T14:47:23Z")

</div>

I don’t have experience with that, but I think that is possible.  
In this article is described that OpenTelemetry collector can use remote write to push metrics to Mimir.  
So in that case Tempo can push to Mimir as well.

> **[Configure the OpenTelemetry Collector to write metrics into Mimir | Grafana...](https://grafana.com/docs/mimir/latest/configure/configure-otel-collector/#remote-write)**
>
> Learn how to write metrics from OpenTelemetry Collector into Mimir

---

<div class="post-metadata">

**Author:** ![shashankjb3](https://avatars.discourse-cdn.com/v4/letter/s/49beb7/32.png) [@shashankjb3](https://community.grafana.com/u/shashankjb3)\
**Post date:** [November 11, 2024, 11:29am UTC](https://community.grafana.com/t/tempo-metrics-generator-not-writing-to-prometheus/109895/10 "2024-11-11T11:29:10Z")

</div>

Yes, we are able to send data to mimir.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/grafana/original/2X/e/ef5308cc58c9e5a13187b5012eb99f406928462f.png) [@system](https://community.grafana.com/u/system)\
**Post date:** [November 11, 2025, 11:29am UTC](https://community.grafana.com/t/tempo-metrics-generator-not-writing-to-prometheus/109895/11 "2025-11-11T11:29:42Z")

</div>

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.