Grafana gossip ring service seems to have wrong IP addresses

[tzaspelfinatix]

Hey there,

I’m new in Grafana and his services. My goal is to implement Grafana Tempo. For this case I the use tempo-distrbuted helm chart.

Now I want to test if all works correctly and I got this error:

 ts=2021-11-12T09:30:46.444315587Z caller=memberlist_logger.go:74 level=warn msg="Was able to connect to tempo-distributed-ingester-0-19d561c3 but other probes failed, network may be misconfigured"                                                                                  │
│ level=warn ts=2021-11-12T09:30:50.437936818Z caller=tcp_transport.go:418 msg="TCPTransport: WriteTo failed" addr=10.4.0.163:7946 err="dial tcp 10.4.0.163:7946: connect: no route to host"

I don’t why the gossip ring point to this IP addresses because they doesn’t exists in the kubernetes cluster.

[joeelliott]

This error is logged when a pod that used to be a part memberlist cluster leaves. It simply means that the memberlist code is attempting to reconnect to a member it used to communicate with and it is not working.

We see this regularly during rollouts.

[tzaspelfinatix]

Thanks for your response. So it sounds like I can ignore this error?

[joeelliott]

Yup. This is ignorable.

[tzaspelfinatix]

Thank your very much!