Grafana gossip ring service seems to have wrong IP addresses

Grafana Tempo
1

Hey there,

I’m new in Grafana and his services. My goal is to implement Grafana Tempo. For this case I the use tempo-distrbuted helm chart.

Now I want to test if all works correctly and I got this error:


 ts=2021-11-12T09:30:46.444315587Z caller=memberlist_logger.go:74 level=warn msg="Was able to connect to tempo-distributed-ingester-0-19d561c3 but other probes failed, network may be misconfigured"                                                                                  │
│ level=warn ts=2021-11-12T09:30:50.437936818Z caller=tcp_transport.go:418 msg="TCPTransport: WriteTo failed" addr=10.4.0.163:7946 err="dial tcp 10.4.0.163:7946: connect: no route to host"

I don’t why the gossip ring point to this IP addresses because they doesn’t exists in the kubernetes cluster.

2

This error is logged when a pod that used to be a part memberlist cluster leaves. It simply means that the memberlist code is attempting to reconnect to a member it used to communicate with and it is not working.

We see this regularly during rollouts.

3

Thanks for your response. So it sounds like I can ignore this error?

4

Yup. This is ignorable.

1 Like
5

Thank your very much!

1 Like
6

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.