Configuring Grafana Tempo and Linkerd for distributed tracing

Grafana Tempo
,
1

Following this article, we can’t seem to get linkerd-jaeger to communicate with the grafana agent with any of the receivers

Our configuration for the grafana-agent is as follows:

kind: ConfigMap
metadata:
  name: grafana-agent-traces
apiVersion: v1
data:
  agent.yaml: |
    server:
      log_level: debug
    traces:
        configs:
          - batch:
                send_batch_size: 1000
                timeout: 5s
            name: default
            receivers:
                jaeger:
                    protocols:
                        grpc: null
                        thrift_binary: null
                        thrift_compact: null
                        thrift_http: null
                    remote_sampling:
                        strategy_file: /etc/agent/strategies.json
                        tls:
                            insecure: true
                opencensus: null
                otlp:
                    protocols:
                        grpc: null
                        http: null
                zipkin: null
            remote_write:
              - endpoint: https://tempo.endpoint.com/otlp
                protocol: http
                retry_on_failure:
                    enabled: false
            scrape_configs:
              - bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
                job_name: kubernetes-pods
                kubernetes_sd_configs:
                  - role: pod
                relabel_configs:
                  - action: replace
                    source_labels:
                      - __meta_kubernetes_namespace
                    target_label: namespace
                  - action: replace
                    source_labels:
                      - __meta_kubernetes_pod_name
                    target_label: pod
                  - action: replace
                    source_labels:
                      - __meta_kubernetes_pod_container_name
                    target_label: container
                tls_config:
                    ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
                    insecure_skip_verify: false
  strategies.json: '{"default_strategy": {"param": 0.001, "type": "probabilistic"}}'

Which we followed this documentation to set up.

We are using linkerd 2.11.4 and deployed it with helm, our values for that deployment are as follows:

## https://github.com/linkerd/linkerd2/blob/stable-2.11.4/jaeger/charts/linkerd-jaeger/values.yaml
linkerd-jaeger:
  collector:
    enabled: false
  jaeger:
    enabled: false
  webhook:
    # -- collector service address for the proxies to send trace data.
    # Points by default to the the linkerd-jaeger collector
    collectorSvcAddr: grafana-agent-traces.monitoring.svc.cluster.local:55678
    # -- service account associated with the collector instance
    collectorSvcAccount: grafana-agent-traces
    logLevel: debug

which we believe lines up with the article posted by Grafana, but is there something we are possibly missing?

Our proxy injector constantly throws this error and we are unsure why the message is “corrupt”

[ 912.146808s] WARN ThreadId(01) opencensus:controller{addr=grafana-agent-traces.monitoring.svc.cluster.local:55678}:endpoint{addr=172.20.0.163:55678}: linkerd_reconnect: Failed to connect error=received corrupt message

2

You can’t forward directly from the clusters to the grafana agent. You need to go via Linkerd’s collector. Try something like this:

## https://github.com/linkerd/linkerd2/blob/stable-2.11.4/jaeger/charts/linkerd-jaeger/values.yaml
linkerd-jaeger:
  jaeger:
    enabled: false

collector:
  config: |
    receivers:
      otlp:
        protocols:
          grpc:
          http:
      opencensus:
      zipkin:
      jaeger:
        protocols:
          grpc:
          thrift_http:
          thrift_compact:
          thrift_binary:
    processors:
      batch:
    extensions:
      health_check:
    exporters:
      otlphttp:
        endpoint: http://grafana-agent-traces.monitoring.svc.cluster.local:4318
    service:
      extensions: [health_check]
      pipelines:
        traces:
          receivers: [otlp,opencensus,zipkin,jaeger]
          processors: [batch]
          exporters: [otlphttp]
    logLevel: debug

You can chose a different format for sending from Linkerd’s collector to your collector.