Grafana agent not pushing traces to Tempo

Grafana Cloud
, ,
1

Hello.

I am evaluating the Grafana Stack for observability and stuck shipping traces to tempo through Grafana Agent.

Setup:

  • Deployed Grafana agent in Kubernetes Cluster and trace_configuration defined in grafana-agent configmap.
  • Tempo & loki datasource being configured in Grafana Cloud

Configuratation file
Deployed Grafana agent from this example agent/production/kubernetes at main · grafana/agent · GitHub

Below is grafana-agent configmap file

apiVersion: v1
kind: ConfigMap
metadata:
  name: grafana-agent
  namespace: monitoring
data:
  agent.yaml: |
    metrics:
      wal_directory: /tmp/grafana-agent-wal
      global:
        scrape_interval: 60s
        external_labels:
          cluster: cloud
      configs:
      - name: integrations
        remote_write:
        - url: "https://prometheus-prod-10-prod-us-central-0.grafana.net/api/prom/push"
          basic_auth:
            username: 342260
            password: "xxxxxx"
        scrape_configs:
        - job_name: integrations/kubernetes/cadvisor
          bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
          kubernetes_sd_configs:
            - role: node
          relabel_configs:
            - replacement: kubernetes.default.svc:443
              target_label: __address__
            - regex: (.+)
              replacement: /api/v1/nodes/$1/proxy/metrics/cadvisor
              source_labels:
                - __meta_kubernetes_node_name
              target_label: __metrics_path__
          scheme: https
          tls_config:
              ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
              insecure_skip_verify: false
              server_name: kubernetes
        - job_name: integrations/kubernetes/kubelet
          bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
          kubernetes_sd_configs:
            - role: node
          relabel_configs:
            - replacement: kubernetes.default.svc:443
              target_label: __address__
            - regex: (.+)
              replacement: /api/v1/nodes/$1/proxy/metrics
              source_labels:
                - __meta_kubernetes_node_name
              target_label: __metrics_path__
          scheme: https
          tls_config:
              ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
              insecure_skip_verify: false
              server_name: kubernetes
    logs:
        configs:
          - clients:
              - url: "https://logs-prod3.grafana.net/loki/api/v1/push"
                basic_auth:
                  username: 170167
                  password: "xxxxxxxxx"
            name: default
            scrape_configs:
              - job_name: kubernetes-pods-name
                kubernetes_sd_configs:
                  - role: pod
                pipeline_stages:
                  - docker: {}
                relabel_configs:
                  - source_labels:
                      - __meta_kubernetes_pod_node_name
                    target_label: __host__
                  - action: labelmap
                    regex: __meta_kubernetes_pod_label_(.+)
                  - action: replace
                    replacement: $1
                    separator: /
                    source_labels:
                      - __meta_kubernetes_namespace
                      - __meta_kubernetes_pod_name
                    target_label: job
                  - action: replace
                    source_labels:
                      - __meta_kubernetes_namespace
                    target_label: namespace
                  - action: replace
                    source_labels:
                      - __meta_kubernetes_pod_name
                    target_label: pod
                  - action: replace
                    source_labels:
                      - __meta_kubernetes_pod_container_name
                    target_label: container
                  - replacement: /var/log/pods/*$1/*.log
                    separator: /
                    source_labels:
                      - __meta_kubernetes_pod_uid
                      - __meta_kubernetes_pod_container_name
                    target_label: __path__
        positions_directory: /tmp/positions


    traces:
      configs:
        - batch:
              send_batch_size: 1000
              timeout: 5s
          name: default
          automatic_logging:
            backend: logs_instance
            logs_instance_name: default
            roots: true
            spans: true
          receivers:
              otlp:
                protocols:
                  grpc: 
                  http: 
          remote_write:
            - basic_auth:
                password: "xxxxxxxx
                username: 1667854
              endpoint: "tempo-us-central1.grafana.net:443"
              insecure: true
              retry_on_failure:
                enabled: false
          scrape_configs:
            - bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
              job_name: kubernetes-pods
              kubernetes_sd_configs:
                - role: pod
              relabel_configs:
                - action: replace
                  source_labels:
                    - __meta_kubernetes_namespace
                  target_label: namespace
                - action: replace
                  source_labels:
                    - __meta_kubernetes_pod_name
                  target_label: pod
                - action: replace
                  source_labels:
                    - __meta_kubernetes_pod_container_name
                  target_label: container
              tls_config:
                  ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
                  insecure_skip_verify: false

Below is the logs of Grafana trace Agent pod.

ts=2022-04-05T07:21:46Z level=info caller=builder/exporters_builder.go:40 msg="Exporter is starting..." component=traces traces_config=default kind=exporter name=otlp/0
ts=2022-04-05T07:21:46Z level=info caller=builder/exporters_builder.go:48 msg="Exporter started." component=traces traces_config=default kind=exporter name=otlp/0
ts=2022-04-05T07:21:46.185604227Z caller=kubernetes.go:284 level=info component="traces service disco" discovery=kubernetes msg="Using pod service account via in-cluster config"
ts=2022-04-05T07:21:46Z level=info caller=builder/pipelines_builder.go:222 msg="Pipeline was built." component=traces traces_config=default name=pipeline name=traces
ts=2022-04-05T07:21:46Z level=info caller=builder/pipelines_builder.go:54 msg="Pipeline is starting..." component=traces traces_config=default name=pipeline name=traces
ts=2022-04-05T07:21:46Z level=info caller=builder/pipelines_builder.go:65 msg="Pipeline is started." component=traces traces_config=default name=pipeline name=traces
ts=2022-04-05T07:21:46Z level=info caller=builder/receivers_builder.go:224 msg="Receiver was built." component=traces traces_config=default kind=receiver name=otlp datatype=traces
ts=2022-04-05T07:21:46Z level=info caller=builder/receivers_builder.go:68 msg="Receiver is starting..." component=traces traces_config=default kind=receiver name=otlp
ts=2022-04-05T07:21:46Z level=info caller=otlpreceiver/otlp.go:68 msg="Starting GRPC server on endpoint 0.0.0.0:4317" component=traces traces_config=default kind=receiver name=otlp
ts=2022-04-05T07:21:46Z level=info caller=otlpreceiver/otlp.go:86 msg="Starting HTTP server on endpoint 0.0.0.0:4318" component=traces traces_config=default kind=receiver name=otlp
ts=2022-04-05T07:21:46Z level=info caller=otlpreceiver/otlp.go:141 msg="Setting up a second HTTP listener on legacy endpoint 0.0.0.0:55681" component=traces traces_config=default kind=receiver name=otlp
ts=2022-04-05T07:21:46Z level=info caller=otlpreceiver/otlp.go:86 msg="Starting HTTP server on endpoint 0.0.0.0:55681" component=traces traces_config=default kind=receiver name=otlp
ts=2022-04-05T07:21:46Z level=info caller=builder/receivers_builder.go:73 msg="Receiver started." component=traces traces_config=default kind=receiver name=otlp
ts=2022-04-05T07:21:46.189779113Z caller=server.go:77 level=info msg="server configuration changed, restarting server"
ts=2022-04-05T07:21:46.190017817Z caller=gokit.go:72 level=info http=[::]:8080 grpc=[::]:9095 msg="server listening on addresses

Expectation:
Grafana agent should pushes traces to tempo to visualize traces generated by application code

2

Hello! I believe the Kubernetes Integration only supports shipping metrics at this time, so logs and traces will need to be shipped from a separate Agent Deployment or Daemonset. We have a quickstart guide for this here, if you’d like to take a look:

In any case, if you’re still experiencing an issue with this please open a support ticket so our team can take a closer look and further assist!