Following this article, we can’t seem to get linkerd-jaeger to communicate with the grafana agent with any of the receivers
Our configuration for the grafana-agent is as follows:
kind: ConfigMap
metadata:
name: grafana-agent-traces
apiVersion: v1
data:
agent.yaml: |
server:
log_level: debug
traces:
configs:
- batch:
send_batch_size: 1000
timeout: 5s
name: default
receivers:
jaeger:
protocols:
grpc: null
thrift_binary: null
thrift_compact: null
thrift_http: null
remote_sampling:
strategy_file: /etc/agent/strategies.json
tls:
insecure: true
opencensus: null
otlp:
protocols:
grpc: null
http: null
zipkin: null
remote_write:
- endpoint: https://tempo.endpoint.com/otlp
protocol: http
retry_on_failure:
enabled: false
scrape_configs:
- bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
job_name: kubernetes-pods
kubernetes_sd_configs:
- role: pod
relabel_configs:
- action: replace
source_labels:
- __meta_kubernetes_namespace
target_label: namespace
- action: replace
source_labels:
- __meta_kubernetes_pod_name
target_label: pod
- action: replace
source_labels:
- __meta_kubernetes_pod_container_name
target_label: container
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
insecure_skip_verify: false
strategies.json: '{"default_strategy": {"param": 0.001, "type": "probabilistic"}}'
Which we followed this documentation to set up.
We are using linkerd 2.11.4 and deployed it with helm, our values for that deployment are as follows:
## https://github.com/linkerd/linkerd2/blob/stable-2.11.4/jaeger/charts/linkerd-jaeger/values.yaml
linkerd-jaeger:
collector:
enabled: false
jaeger:
enabled: false
webhook:
# -- collector service address for the proxies to send trace data.
# Points by default to the the linkerd-jaeger collector
collectorSvcAddr: grafana-agent-traces.monitoring.svc.cluster.local:55678
# -- service account associated with the collector instance
collectorSvcAccount: grafana-agent-traces
logLevel: debug
which we believe lines up with the article posted by Grafana, but is there something we are possibly missing?
Our proxy injector constantly throws this error and we are unsure why the message is “corrupt”
[ 912.146808s] WARN ThreadId(01) opencensus:controller{addr=grafana-agent-traces.monitoring.svc.cluster.local:55678}:endpoint{addr=172.20.0.163:55678}: linkerd_reconnect: Failed to connect error=received corrupt message