Hello! I’ve been banging my head on this all day now. I’ve got a Tempo POC deployment set up in AWS under Fargate, based off my in-production Loki deployment. I have the temp distributor configured to receive jaeger thrift over http. I’m using Grafana 7.5.0 so I started pulling ‘grafana/tempo:latest’ from dockerhub after figuring out that the 0.6.0 release didn’t work with Grafana trace queries.
distributor: receivers: jaeger: protocols: thrift_http: endpoint: 0.0.0.0:14268
And in the ingester is the basic stock config.
ingester: lifecycler: ring: replication_factor: 1 trace_idle_period: 30s max_block_bytes: 1_000_000 max_block_duration: 1h
Both have the same server block:
server: http_listen_address: 0.0.0.0 grpc_listen_address: 0.0.0.0 http_listen_port: 3100 grpc_listen_port: 9095
And they’re using the memberlist on 7649. I’m using the docker-compose example to generate traces and throw them at my distributor, but all I’m getting is this error in the distributor logs:
level=error ts=2021-03-28T00:47:41.97961568Z caller=log.go:27 msg=“pusher failed to consume trace data” err=“rpc error: code = Unimplemented desc = unknown service tempopb.Pusher”
Checking the troubleshooting page in the documentation I followed up on the metrics, the distributor records a few hundred traces received, but that they all failed. The ingester shows no indication of any activity. They’re all running from a common config file, all with the same ports exposed and allowed over security group.
Anything else I can double check?