Tempo distributed installing successfully but traces not pushing in grafana

[sbhushankr]

We have installed the tempo distributed on EKS 1.23 version and all the pods are showing as running

But when we try to push the traces then it is giving the connection refused.

Regards,
Shashi

[sbhushankr]

Hi,

We are using the below values.yaml file to deploy the same. Are we missing something on this.

---

global:
clusterDomain: ‘cluster.local’
gateway:
enabled: true
traces:
otlp:
http:
enabled: true
grpc:
enabled: true
querier:
extraEnv:
- name: MY_POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
extraArgs:
- -config.expand-env=true
distributor:
config:
log_received_spans:
enabled: true
extraEnv:
- name: MY_POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
extraArgs:
- -config.expand-env=true
compactor:
extraEnv:
- name: MY_POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
extraArgs:
- -config.expand-env=true
ingester:
extraEnv:
- name: MY_POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
extraArgs:
- -config.expand-env=true
config: |
multitenancy_enabled: false
compactor:
compaction:
block_retention: 48h
ring:
heartbeat_timeout: 5m
kvstore:
store: memberlist
distributor:
log_received_traces: true
receivers:
jaeger:
protocols:
grpc:
endpoint: 0.0.0.0:14250
thrift_binary:
endpoint: 0.0.0.0:6832
thrift_compact:
endpoint: 0.0.0.0:6831
thrift_http:
endpoint: 0.0.0.0:14268
log_received_spans:
enabled: true
querier:
frontend_worker:
frontend_address: {{ include “tempo.resourceName” (dict “ctx” . “component” “query-frontend”) }}:9095
ingester:
lifecycler:
join_after: 30s
ring:
replication_factor: 1
heartbeat_timeout: 1m
kvstore:
store: memberlist
tokens_file_path: /var/tempo/tokens.json
memberlist:
bind_addr:
- ${MY_POD_IP}
abort_if_cluster_join_fails: false
randomize_node_name: false
leave_timeout: 10s
pull_push_interval: 10s
rejoin_interval: 15m
bind_port: 7946
join_members:
- {{ include “tempo.fullname” . }}-gossip-ring
overrides:
per_tenant_override_config: /conf/overrides.yaml
server:
http_listen_port: 3100
storage:
trace:
backend: s3
s3:
bucket: XXXXXXXXXX-mulesoft-eks-tempo-logs-dev
endpoint: s3.us-west-2.amazonaws.com
insecure: false
pool:
queue_depth: 2000
wal:
path: /var/tempo/wal
memcached:
consistent_hash: true
host: a-tempo-distributed-memcached
service: memcached-client
timeout: 500ms

Regards,
Shashi

[mdisibio]

Hi it looks like only the jaeger receiver is enabled. Can you try enabling the otlp receivers? In this section below, documentation here

distributor:
  receivers:
    jaeger:
      protocols:
       grpc:
         endpoint: 0.0.0.0:14250
       thrift_binary:
         endpoint: 0.0.0.0:6832
       thrift_compact:
         endpoint: 0.0.0.0:6831
       thrift_http:
         endpoint: 0.0.0.0:14268