Grafana Agent to Grafana Tempo

Grafana Cloud
,
1
  • What Grafana version and what operating system are you using?

Grafana Cloud

  • What are you trying to achieve?

We want to collect traces in our application and post them to Grafana Tempo

  • How are you trying to achieve it?

We have a rust application which collects traces and pushes them to a locally running agent. For that we are using grafana-agent (docker container grafana/agent:v0.25.0)

  • What happened?

The agent prints errors that it can’t post the traces to Temp, e.g.

data-agent-1  | ts=2022-07-11T05:31:22Z level=info caller=exporterhelper/queued_retry.go:218 msg="Exporting failed. Will retry the request after interval." component=traces traces_config=default kind=exporter name=otlphttp/0 error="error exporting items, request to https://tempo-prod-03-au-southeast-0.grafana.net/tempo/v1/traces responded with HTTP Status Code 404" interval=3.452151555s
  • What did you expect to happen?

To see the traces in Grafana Tempo :slight_smile:

  • Can you copy/paste the configuration(s) that you are having problems with?

Agent config:

server:
  log_level: debug

traces:
  configs:
    - name: default
      receivers:
        otlp:
          protocols:
            grpc:
              endpoint: "0.0.0.0:4317"
      remote_write:
#        - endpoint: tempo-prod-03-au-southeast-0.grafana.net:443
        - endpoint: https://tempo-prod-03-au-southeast-0.grafana.net/tempo
#          insecure: true  # only add this if TLS is not required
#          protocol: "grpc"
          protocol: "http"
          basic_auth:
            username: USERNAME
            password: API_KEY

      batch:
        timeout: 5s
        send_batch_size: 100

As you can see in the config above, we tried different configurations

  • Did you receive any errors in the Grafana UI or in related logs? If so, please tell us exactly what they were.

Nothing

  • Did you follow any online instructions? If so, what is the URL?

Followed the instruction in this Documentation: Run Grafana Agent on Docker | Grafana Agent documentation

Thanks for your help.

1 Like