Exemplars not working

Grafana Prometheus
1

  • What Grafana version and what operating system are you using?
    v11.6.0 (d2fdff9ee4)

  • What are you trying to achieve?
    I’m trying to scrape metrics with trace IDs and store them in Mimir to visualize exemplars in a Grafana dashboard panel.

  • How are you trying to achieve it?
    I have a set of docker containers (Grafana, Prometheus, Mimir, Tempo, OpenTelemetry Collector and Loki) and a little hello world Golang application that is emitting metrics with the Span & Trace information embedded into the a metric (counter). I can see the metrics:

curl -s "http://localhost:9464/metrics" | head -n10
# HELP sre_my_counter_total
# TYPE sre_my_counter_total counter
sre_my_counter_total{job="metal-app",operacao="oi"} 80
sre_my_counter_total{job="metal-app",operacao="oi",span_id="01c2cf445ba75703",trace_id="b945f37a1d57ad1f14a39baaf95f174e"} 1
sre_my_counter_total{job="metal-app",operacao="oi",span_id="0244deb2e70ae91c",trace_id="d5c1c215b10aeae6e842e06ac3013fba"} 1

I initialized Prometheus with the --enable-feature=exemplar-storage flag.
I also added the following remote_write configuration to Prometheus to send the metrics over to Mimir:

remote_write:
  - url: "http://mimir:9009/api/v1/push"
    send_exemplars: true

I also added the max_global_exemplars_per_user parameter to my mimir-config.yml:

limits:
  max_global_exemplars_per_user: 100000

Then, in my local Grafana, I just can’t make the exemplars show up. When I mouseover the datapoints I can see the span_id and trace_id labels in the metric but the fancy exemplars dialog just doesn’t show up. I have configured the Tempo datasource, confirmed the traces are flowing, and even my Mimir data source has the Exemplar’s configuration set against “internal link = Tempo” with the “label name = trace_id”. How do I enable the exemplars visualization on my dashboard panel?

  • What happened?
    I just can’t make the exemplars show up. I tried to toggle the “Exemplars” option under the panel’s query options, but that did not help.

  • What did you expect to happen?
    I just want to see the exemplars floating dialog / modal with the metric information containing a link to the trace ID.

  • Can you copy/paste the configuration(s) that you are having problems with?
    I have the experiment’s artifacts available here:
    Comparing main...experiment-with-exemplars · themarcelor/metal-app · GitHub

  • Did you receive any errors in the Grafana UI or in related logs? If so, please tell us exactly what they were.
    I didn’t get any errors in the Grafana UI.

  • Did you follow any online instructions? If so, what is the URL?
    I followed some of these steps here:
    https://grafana.com/docs/mimir/latest/manage/use-exemplars/before-you-begin/
    and the rest of the configuration I put together through additional research after trying several different things.