Hello.
I am evaluating the Grafana Stack for observability and stuck shipping traces to tempo through Grafana Agent.
Setup:
- Deployed Grafana agent in Kubernetes Cluster and trace_configuration defined in grafana-agent configmap.
- Tempo & loki datasource being configured in Grafana Cloud
Configuratation file
Deployed Grafana agent from this example agent/production/kubernetes at main · grafana/agent · GitHub
Below is grafana-agent configmap file
apiVersion: v1
kind: ConfigMap
metadata:
name: grafana-agent
namespace: monitoring
data:
agent.yaml: |
metrics:
wal_directory: /tmp/grafana-agent-wal
global:
scrape_interval: 60s
external_labels:
cluster: cloud
configs:
- name: integrations
remote_write:
- url: "https://prometheus-prod-10-prod-us-central-0.grafana.net/api/prom/push"
basic_auth:
username: 342260
password: "xxxxxx"
scrape_configs:
- job_name: integrations/kubernetes/cadvisor
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
kubernetes_sd_configs:
- role: node
relabel_configs:
- replacement: kubernetes.default.svc:443
target_label: __address__
- regex: (.+)
replacement: /api/v1/nodes/$1/proxy/metrics/cadvisor
source_labels:
- __meta_kubernetes_node_name
target_label: __metrics_path__
scheme: https
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
insecure_skip_verify: false
server_name: kubernetes
- job_name: integrations/kubernetes/kubelet
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
kubernetes_sd_configs:
- role: node
relabel_configs:
- replacement: kubernetes.default.svc:443
target_label: __address__
- regex: (.+)
replacement: /api/v1/nodes/$1/proxy/metrics
source_labels:
- __meta_kubernetes_node_name
target_label: __metrics_path__
scheme: https
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
insecure_skip_verify: false
server_name: kubernetes
logs:
configs:
- clients:
- url: "https://logs-prod3.grafana.net/loki/api/v1/push"
basic_auth:
username: 170167
password: "xxxxxxxxx"
name: default
scrape_configs:
- job_name: kubernetes-pods-name
kubernetes_sd_configs:
- role: pod
pipeline_stages:
- docker: {}
relabel_configs:
- source_labels:
- __meta_kubernetes_pod_node_name
target_label: __host__
- action: labelmap
regex: __meta_kubernetes_pod_label_(.+)
- action: replace
replacement: $1
separator: /
source_labels:
- __meta_kubernetes_namespace
- __meta_kubernetes_pod_name
target_label: job
- action: replace
source_labels:
- __meta_kubernetes_namespace
target_label: namespace
- action: replace
source_labels:
- __meta_kubernetes_pod_name
target_label: pod
- action: replace
source_labels:
- __meta_kubernetes_pod_container_name
target_label: container
- replacement: /var/log/pods/*$1/*.log
separator: /
source_labels:
- __meta_kubernetes_pod_uid
- __meta_kubernetes_pod_container_name
target_label: __path__
positions_directory: /tmp/positions
traces:
configs:
- batch:
send_batch_size: 1000
timeout: 5s
name: default
automatic_logging:
backend: logs_instance
logs_instance_name: default
roots: true
spans: true
receivers:
otlp:
protocols:
grpc:
http:
remote_write:
- basic_auth:
password: "xxxxxxxx
username: 1667854
endpoint: "tempo-us-central1.grafana.net:443"
insecure: true
retry_on_failure:
enabled: false
scrape_configs:
- bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
job_name: kubernetes-pods
kubernetes_sd_configs:
- role: pod
relabel_configs:
- action: replace
source_labels:
- __meta_kubernetes_namespace
target_label: namespace
- action: replace
source_labels:
- __meta_kubernetes_pod_name
target_label: pod
- action: replace
source_labels:
- __meta_kubernetes_pod_container_name
target_label: container
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
insecure_skip_verify: false
Below is the logs of Grafana trace Agent pod.
ts=2022-04-05T07:21:46Z level=info caller=builder/exporters_builder.go:40 msg="Exporter is starting..." component=traces traces_config=default kind=exporter name=otlp/0
ts=2022-04-05T07:21:46Z level=info caller=builder/exporters_builder.go:48 msg="Exporter started." component=traces traces_config=default kind=exporter name=otlp/0
ts=2022-04-05T07:21:46.185604227Z caller=kubernetes.go:284 level=info component="traces service disco" discovery=kubernetes msg="Using pod service account via in-cluster config"
ts=2022-04-05T07:21:46Z level=info caller=builder/pipelines_builder.go:222 msg="Pipeline was built." component=traces traces_config=default name=pipeline name=traces
ts=2022-04-05T07:21:46Z level=info caller=builder/pipelines_builder.go:54 msg="Pipeline is starting..." component=traces traces_config=default name=pipeline name=traces
ts=2022-04-05T07:21:46Z level=info caller=builder/pipelines_builder.go:65 msg="Pipeline is started." component=traces traces_config=default name=pipeline name=traces
ts=2022-04-05T07:21:46Z level=info caller=builder/receivers_builder.go:224 msg="Receiver was built." component=traces traces_config=default kind=receiver name=otlp datatype=traces
ts=2022-04-05T07:21:46Z level=info caller=builder/receivers_builder.go:68 msg="Receiver is starting..." component=traces traces_config=default kind=receiver name=otlp
ts=2022-04-05T07:21:46Z level=info caller=otlpreceiver/otlp.go:68 msg="Starting GRPC server on endpoint 0.0.0.0:4317" component=traces traces_config=default kind=receiver name=otlp
ts=2022-04-05T07:21:46Z level=info caller=otlpreceiver/otlp.go:86 msg="Starting HTTP server on endpoint 0.0.0.0:4318" component=traces traces_config=default kind=receiver name=otlp
ts=2022-04-05T07:21:46Z level=info caller=otlpreceiver/otlp.go:141 msg="Setting up a second HTTP listener on legacy endpoint 0.0.0.0:55681" component=traces traces_config=default kind=receiver name=otlp
ts=2022-04-05T07:21:46Z level=info caller=otlpreceiver/otlp.go:86 msg="Starting HTTP server on endpoint 0.0.0.0:55681" component=traces traces_config=default kind=receiver name=otlp
ts=2022-04-05T07:21:46Z level=info caller=builder/receivers_builder.go:73 msg="Receiver started." component=traces traces_config=default kind=receiver name=otlp
ts=2022-04-05T07:21:46.189779113Z caller=server.go:77 level=info msg="server configuration changed, restarting server"
ts=2022-04-05T07:21:46.190017817Z caller=gokit.go:72 level=info http=[::]:8080 grpc=[::]:9095 msg="server listening on addresses
Expectation:
Grafana agent should pushes traces to tempo to visualize traces generated by application code