# How enable Service Graph with Tempo

**URL:** <https://community.grafana.com/t/how-enable-service-graph-with-tempo/87715>\
**Category:** Grafana Tempo\
**Created:** [May 16, 2023, 10:06am UTC](https://community.grafana.com/t/how-enable-service-graph-with-tempo/87715 "2023-05-16T10:06:56Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![kiuby88freak](https://avatars.discourse-cdn.com/v4/letter/k/eb8c5e/32.png) [@kiuby88freak](https://community.grafana.com/u/kiuby88freak)\
**Post date:** [May 16, 2023, 10:06am UTC](https://community.grafana.com/t/how-enable-service-graph-with-tempo/87715/1 "2023-05-16T10:06:56Z")

</div>

I cannot configure Grafana Tempo to produce span-metrics.  
I have using spring applications with OpenTelemetryAgent, and I have deployed Grafana, Prometheus and Tempo using Docker Compose.  
Docker compose

```auto
  grafana:
    image: grafana/grafana:9.5.2 # this is for tempo 2
    container_name: grafana
    hostname: grafana 
    depends_on:
      tempo:
        condition: service_healthy
      prometheus:
        condition: service_healthy
    volumes:
      - ./config/grafana-bootstrap.ini:/etc/grafana/grafana.ini
      - ./config/grafana-datasources.yaml:/etc/grafana/provisioning/datasources/datasources.yaml
    environment:
      - GF_AUTH_ANONYMOUS_ENABLED=true
      - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
      - GF_AUTH_DISABLE_LOGIN_FORM=true
    ports:
      - "3000:3000"
    healthcheck:
      interval: 5s
      retries: 10
      test: wget --no-verbose --tries=1 --spider http://grafana.staged-by-discourse.com || exit 1

  prometheus:
    image: prom/prometheus:v2.41.0
    container_name: prometheus
    hostname: prometheus
    command:
      - --config.file=/etc/prometheus.yaml
      - --web.enable-remote-write-receiver
      - --enable-feature=exemplar-storage
    volumes:
      - ./config/prometheus.yaml:/etc/prometheus.yaml
    ports:
      - "9090:9090"
    healthcheck:
      interval: 5s
      retries: 10
      test: wget --no-verbose --tries=1 --spider http://localhost:9090/status || exit 1

  tempo:
    image: grafana/tempo:1.5.0
    command: ["-search.enabled=true", "-config.file=/etc/tempo.yaml"]
    container_name: tempo
    hostname: tempo
    volumes:
      - ./config/tempo-config.yaml:/etc/tempo.yaml
    ports:
      - "3200:3200"
      - "4317:4317"
      - "4318:4318"
    expose:
      - "42168"
    healthcheck:
      interval: 5s
      retries: 10
      test: wget --no-verbose --tries=1 --spider http://localhost:3200/status || exit 1

```

Here you can see `tempo-config.yaml` file

```auto
server:
  http_listen_port: 3200

distributor:
  search_tags_deny_list:
    - "instance"
    - "version"
  receivers:
    jaeger:
      protocols:
        thrift_http:
        grpc:
        thrift_binary:
        thrift_compact:
    zipkin:
    otlp:
      protocols:
        http:
        grpc:
    opencensus:

ingester:
  trace_idle_period: 10s
  max_block_bytes: 1_000_000
  max_block_duration: 5m

compactor:
  compaction:
    compaction_window: 1h
    max_block_bytes: 100_000_000
    block_retention: 1h
    compacted_block_retention: 10m

storage:
  trace:
    backend: local
    block:
      bloom_filter_false_positive: .05
      index_downsample_bytes: 1000
      encoding: zstd
    wal:
      path: /tmp/tempo/wal
      encoding: snappy
    local:
      path: /tmp/tempo/blocks
    pool:
      max_workers: 100
      queue_depth: 10000

metrics_generator:
  registry:
    collection_interval: 5s   
    external_labels:
      source: tempo
      cluster: docker-compose
  storage:
    path: /tmp/tempo/generator/wal
    remote_write:
      - url: http://prometheus:9090/api/v1/write
        send_exemplars: true

overrides:
  metrics_generator_processors:
    - service-graphs
    - span-metrics

```

Here you can see `prometheus.yaml`

```auto
global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
- job_name: 'collector'
  scrape_interval: 15s
  static_configs:
    - targets: ['collector:6666']

```

Finally, `grafana-datasources.yaml` and `grafana-bootstrap.ini`

```auto
apiVersion: 1

datasources:
- name: Prometheus
  type: prometheus
  access: proxy
  orgId: 1
  url: http://prometheus:9090
  basicAuth: false
  isDefault: false
  version: 1
  editable: true
  jsonData:
    httpMethod: GET
- name: Tempo
  type: tempo
  access: proxy
  orgId: 1
  url: http://tempo:3200
  jsonData:
    httpMethod: GET
    serviceMap:
      datasourceUid: 'Prometheus'
  basicAuth: false
  isDefault: true
  version: 1
  editable: true
  apiVersion: 1
  uid: tempo

```

```auto
[feature_toggles]
enable = tempoSearch tempoBackendSearch tempoServiceGraph

```

I have also enabled `Service Graph ` in the the `Tempo DataSource` configuration in Grafana, as you can see below.

 ![Screenshot 2023-05-16 at 12.02.24](https://us1.discourse-cdn.com/grafana/original/3X/7/6/76f754d673da7993f44639faa3072f498aa960c7.png)

I can see the traces (as you can see below) but I the Service Graph says there are not data available. I am using Tempo 1.5.X version. I tried to use 2.1.X (migrating the configuration) and I got the same result.

 ![Screenshot 2023-05-16 at 12.03.13](https://us1.discourse-cdn.com/grafana/original/3X/3/3/33f12f5ed9cf9c8ff34c295964dab68afd255926.png)

I thing I am missing something (maybe in the prometheus configuration), but I do not know how fix it.

Can you help me?  
Thanks

---

<div class="post-metadata">

**Author:** ![surajsidh](https://sea2.discourse-cdn.com/grafana/user_avatar/community.grafana.com/surajsidh/32/15860_2.png) [@surajsidh](https://community.grafana.com/u/surajsidh)\
**Post date:** [May 16, 2023, 10:18am UTC](https://community.grafana.com/t/how-enable-service-graph-with-tempo/87715/2 "2023-05-16T10:18:14Z")

</div>

can you check if you have all [spam metrics](https://grafana.com/docs/tempo/latest/metrics-generator/span_metrics/#metrics) in your promethous?

you can use explore to check for `traces_spanmetrics_latency`, `traces_spanmetrics_calls_total` and `traces_spanmetrics_size_total` metrics?

also see [metrics-generator config docs](https://grafana.com/docs/tempo/latest/configuration/#metrics-generator) for details of metrics-generator config.

---

<div class="post-metadata">

**Author:** ![kiuby88freak](https://avatars.discourse-cdn.com/v4/letter/k/eb8c5e/32.png) [@kiuby88freak](https://community.grafana.com/u/kiuby88freak)\
**Post date:** [May 17, 2023, 7:24am UTC](https://community.grafana.com/t/how-enable-service-graph-with-tempo/87715/3 "2023-05-17T07:24:03Z")

</div>

Thanks for the response @surajsidh

The `spanmetrics` metrics are missing in Prometheus. Maybe the issue is in the `span_metrics.storage` [config](https://grafana.com/docs/tempo/v1.5.x/configuration/#metrics-generator), but I have enabled [Remote Writes](https://prometheus.io/docs/prometheus/latest/feature_flags/#remote-write-receiver) according to the documentation, see docker-compose above

```auto
      - --web.enable-remote-write-receiver

```

Tempo is configured to write metrics in the endpoint (see tempo-config.yaml above):

```auto
    remote_write:
      - url: http://prometheus:9090/api/v1/write
        send_exemplars: true

```

I cannot see any related message in the logs.  
(I have simplified a bit the configuration in my post).

Do you know if there is an example of Tempo configuration (using K8S o Docker Compose) with the `span_metrics` running?

Many thanks

---

<div class="post-metadata">

**Author:** ![kiuby88freak](https://avatars.discourse-cdn.com/v4/letter/k/eb8c5e/32.png) [@kiuby88freak](https://community.grafana.com/u/kiuby88freak)\
**Post date:** [May 17, 2023, 7:38am UTC](https://community.grafana.com/t/how-enable-service-graph-with-tempo/87715/4 "2023-05-17T07:38:24Z")

</div>

Solved

I forgot the property in the `tempo-config.yaml`.

```plaintext
metrics_generator_enabled: true

```

---

<div class="post-metadata">

**Author:** ![virajgamage](https://avatars.discourse-cdn.com/v4/letter/v/3da27b/32.png) [@virajgamage](https://community.grafana.com/u/virajgamage)\
**Post date:** [August 9, 2023, 8:40am UTC](https://community.grafana.com/t/how-enable-service-graph-with-tempo/87715/5 "2023-08-09T08:40:59Z")

</div>

Hi, where to add this property, I am having the same problem.  
I added it in the root level of the yaml file, but it is still not working.

---

<div class="post-metadata">

**Author:** ![stratusss](https://avatars.discourse-cdn.com/v4/letter/s/4bbf92/32.png) [@stratusss](https://community.grafana.com/u/stratusss)\
**Post date:** [August 14, 2023, 2:36pm UTC](https://community.grafana.com/t/how-enable-service-graph-with-tempo/87715/6 "2023-08-14T14:36:57Z")

</div>

I wanted to chime in in case someone else had this issue similar to me.

I am setting this up on VMs instead of K8S or other container platforms.

I had the following in my `temp/config.yaml` file:

```auto
metrics_generator:
  registry:
    external_labels:
      source: tempo
      cluster: linux-microservices
  storage:
    path: /tmp/tempo/generator/wal
    remote_write:
    - url: http://x.x.x.x:9090/api/v1/write
      send_exemplars: true

```

I had grafana and prometheus setup on their vms as well. I was seeing trace data in grafana from tempo but I wasn’t seeing the service graphs.

In my case it was that I needed to add `--enable-feature=remote-write-receiver ` and `--enable-feature=exemplar-storage` to my prometheus systemd file. My full prometheus systemd file is as follows:

```auto
[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
--config.file /etc/prometheus/prometheus.yaml \
--storage.tsdb.path /var/lib/prometheus/ \
--web.console.templates=/etc/prometheus/consoles \
--web.console.libraries=/etc/prometheus/console_libraries \
--enable-feature=remote-write-receiver \
--enable-feature=exemplar-storage

[Install]
WantedBy=multi-user.target

```

Did a `systemctl daemon-reload && systemctl restart prometheus` and the service graphs started showing in grafana

for the record I am using the following for my grafana datasource as I wasn’t able to figure out how to configure tempo as a datasource in the grafana web UI:

```auto
apiVersion: 1

datasources:
- name: Prometheus
  type: prometheus
  uid: prometheus
  access: proxy
  orgId: 1
  url: http://localhost:9090
  basicAuth: false
  isDefault: false
  version: 1
  editable: false
  jsonData:
    httpMethod: GET
- name: Tempo
  type: tempo
  access: proxy
  orgId: 1
  url: http://x.x.x.x:3200
  basicAuth: false
  isDefault: true
  version: 1
  editable: false
  apiVersion: 1
  uid: tempo
  jsonData:
    httpMethod: GET
    serviceMap:
      datasourceUid: prometheus

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/grafana/original/2X/e/ef5308cc58c9e5a13187b5012eb99f406928462f.png) [@system](https://community.grafana.com/u/system)\
**Post date:** [August 13, 2024, 2:37pm UTC](https://community.grafana.com/t/how-enable-service-graph-with-tempo/87715/7 "2024-08-13T14:37:05Z")

</div>

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.