Build cluster failed

lgt · August 5, 2021, 1:34am

When I start the node by docker ,it show msg like this. What’s the problem here, or how can I find the detail error somewhere? thanks

dannykopping · August 5, 2021, 6:55am

Hi @lgt

This appears to be a networking issue on your side, relating to memberlist.
Here is an issue which appears related to your question, with some debugging instructions:

github.com/hashicorp/consul

memberlist: Was able to connect to xxx but other probes failed, network may be misconfigured

opened 09:36AM - 07 Jan 19 UTC

closed 10:53PM - 15 Jan 19 UTC

TwitchChen

I'm trying to find out what caused this problem, but i'm failed. the logs ``…` 2019/01/07 17:25:30 [DEBUG] memberlist: Failed ping: hkam30008 (timeout reached) 2019/01/07 17:25:31 [DEBUG] memberlist: Stream connection from=100.69.28.7:63057 2019/01/07 17:25:31 [DEBUG] memberlist: Failed ping: hkam30162 (timeout reached) 2019/01/07 17:25:32 [WARN] memberlist: Was able to connect to hkam30162 but other probes failed, network may be misconfigured 2019/01/07 17:25:32 [DEBUG] memberlist: Failed ping: hkam30009 (timeout reached) 2019/01/07 17:25:32 [DEBUG] agent: Node info in sync 2019/01/07 17:25:33 [WARN] memberlist: Was able to connect to hkam30009 but other probes failed, network may be misconfigured 2019/01/07 17:25:33 [DEBUG] memberlist: Stream connection from=100.69.28.1:42918 2019/01/07 17:25:33 [DEBUG] memberlist: Stream connection from=100.69.28.8:63323 2019/01/07 17:25:34 [DEBUG] memberlist: Stream connection from=100.69.28.16:47760 2019/01/07 17:25:37 [DEBUG] memberlist: Failed ping: hkam30059 (timeout reached) 2019/01/07 17:25:39 [DEBUG] memberlist: Stream connection from=100.69.31.156:24380 2019/01/07 17:25:39 [DEBUG] memberlist: Failed ping: hkam30020 (timeout reached) 2019/01/07 17:25:40 [WARN] memberlist: Was able to connect to hkam30020 but other probes failed, network may be misconfigured 2019/01/07 17:25:41 [DEBUG] memberlist: Failed ping: hkam30163 (timeout reached) 2019/01/07 17:25:42 [WARN] memberlist: Was able to connect to hkam30163 but other probes failed, network may be misconfigured ``` the **consul info** ``` agent: check_monitors = 0 check_ttls = 0 checks = 0 services = 0 build: prerelease = revision = 75ca2ca version = 0.9.2 consul: bootstrap = false known_datacenters = 4 leader = false leader_addr = 100.69.88.1:8300 server = true raft: applied_index = 6271351 commit_index = 6271351 fsm_pending = 0 last_contact = 6.756206ms last_log_index = 6271351 last_log_term = 16 last_snapshot_index = 6265248 last_snapshot_term = 14 latest_configuration = [{Suffrage:Voter ID:100.69.24.1:8300 Address:100.69.24.1:8300} {Suffrage:Voter ID:100.69.24.2:8300 Address:100.69.24.2:8300} {Suffrage:Voter ID:100.69.24.3:8300 Address:100.69.24.3:8300} {Suffrage:Voter ID:100.69.88.2:8300 Address:100.69.88.2:8300} {Suffrage:Voter ID:100.69.88.1:8300 Address:100.69.88.1:8300} {Suffrage:Voter ID:100.69.88.3:8300 Address:100.69.88.3:8300}] latest_configuration_index = 5935917 num_peers = 5 protocol_version = 2 protocol_version_max = 3 protocol_version_min = 0 snapshot_version_max = 1 snapshot_version_min = 0 state = Follower term = 16 runtime: arch = amd64 cpu_count = 40 goroutines = 141 max_procs = 40 os = linux version = go1.8.3 serf_lan: coordinate_resets = 0 encrypted = false event_queue = 0 event_time = 6 failed = 0 health_score = 0 intent_queue = 0 left = 0 member_time = 1208 members = 41 query_queue = 0 query_time = 1 serf_wan: coordinate_resets = 0 encrypted = false event_queue = 0 event_time = 1 failed = 0 health_score = 0 intent_queue = 0 left = 0 member_time = 45 members = 18 query_queue = 0 query_time = 1 ``` the **members** ``` Node Address Status Type Build Protocol DC hkam30005 100.69.28.1:8301 alive client 0.9.2 2 hk1 hkam30006 100.69.28.2:8301 alive client 0.9.2 2 hk1 hkam30007 100.69.28.3:8301 alive client 0.9.2 2 hk1 hkam30008 100.69.28.4:8301 alive client 0.9.2 2 hk1 hkam30009 100.69.28.5:8301 alive client 0.9.2 2 hk1 hkam30010 100.69.28.6:8301 alive client 0.9.2 2 hk1 hkam30011 100.69.28.7:8301 alive client 0.9.2 2 hk1 hkam30012 100.69.28.8:8301 alive client 0.9.2 2 hk1 hkam30013 100.69.24.1:8301 alive server 0.9.2 2 hk1 hkam30014 100.69.24.2:8301 alive server 0.9.2 2 hk1 hkam30015 100.69.24.3:8301 alive server 0.9.2 2 hk1 hkam30019 100.69.28.12:8301 alive client 0.9.2 2 hk1 hkam30020 100.69.28.13:8301 alive client 0.9.2 2 hk1 hkam30021 100.69.28.14:8301 alive client 0.9.2 2 hk1 hkam30022 100.69.28.15:8301 alive client 0.9.2 2 hk1 hkam30023 100.69.28.16:8301 alive client 0.9.2 2 hk1 hkam30058 100.69.31.150:8301 alive client 0.9.2 2 hk1 hkam30059 100.69.31.151:8301 alive client 0.9.2 2 hk1 hkam30162 100.69.31.156:8301 alive client 0.9.2 2 hk1 hkam30163 100.69.31.157:8301 alive client 0.9.2 2 hk1 hkbm30029 100.69.88.1:8301 alive server 0.9.2 2 hk1 hkbm30030 100.69.88.2:8301 alive server 0.9.2 2 hk1 hkbm30031 100.69.88.3:8301 alive server 0.9.2 2 hk1 hkbm30034 100.69.92.150:8301 alive client 0.9.2 2 hk1 hkbm30035 100.69.92.151:8301 alive client 0.9.2 2 hk1 hkbm30036 100.69.92.152:8301 alive client 0.9.2 2 hk1 hkbm30037 100.69.92.153:8301 alive client 0.9.2 2 hk1 hkbm30038 100.69.92.154:8301 alive client 0.9.2 2 hk1 hkbm30039 100.69.92.155:8301 alive client 0.9.2 2 hk1 hkbm30040 100.69.92.156:8301 alive client 0.9.2 2 hk1 hkbm30041 100.69.92.157:8301 alive client 0.9.2 2 hk1 hkbm30042 100.69.92.158:8301 alive client 0.9.2 2 hk1 hkbm30043 100.69.92.159:8301 alive client 0.9.2 2 hk1 hkbm30044 100.69.92.160:8301 alive client 0.9.2 2 hk1 hkbm30045 100.69.92.161:8301 alive client 0.9.2 2 hk1 hkbm30046 100.69.92.162:8301 alive client 0.9.2 2 hk1 hkbm30048 100.69.92.163:8301 alive client 0.9.2 2 hk1 hkbm30061 100.69.92.164:8301 alive client 0.9.2 2 hk1 hkbm30062 100.69.92.165:8301 alive client 0.9.2 2 hk1 hkbm30063 100.69.92.166:8301 alive client 0.9.2 2 hk1 hkbm30064 100.69.92.167:8301 alive client 0.9.2 2 hk1 ``` the **client config** ``` { "datacenter": "HK1", "data_dir": "/pacloud/ccmp/base/ccmp/data", "log_level": "DEBUG", "node_name": "hkbm30064", "server": false, "client_addr":"0.0.0.0", "bind_addr":"100.69.92.167", "retry_join":["100.69.24.1","100.69.24.2","100.69.24.3","100.69.88.1","100.69.88.2","100.69.88.3"] } ```

lgt · August 5, 2021, 8:16am

Thanks for help, I think is not the network problem , because all the firewall is closed ,and I have try to ping from each node and it’s ok. Could you please give me more advice , thanks~
Here is my config.yaml:

and my docker-compose.yaml:

dannykopping · August 5, 2021, 8:35am

I think is not the network problem , because all the firewall is closed ,and I have try to ping from each node and it’s ok

Pings will only tell if your network can pass around ICMP packets; this is unrelated to what your problem is here.

Did you follow the debugging steps provided in that issue?
Are all the required ports accessible, etc?

lgt · August 6, 2021, 12:48am

I have tried telnet 7946 and it’s ok. Is there any more required ports I need to test?

lgt · August 6, 2021, 12:56am

And how can I debug it because loki is running on the server. I’m not familiar with the programing language in that issue show. I guess it’s like a timeout fallback?

system · August 6, 2022, 12:56am

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Loki component="memberlist TCPTransport" msg="failed to read message type" err=EOF Grafana Loki loki , bugreport	3	1596	February 16, 2025
Grafana gossip ring service seems to have wrong IP addresses Grafana Tempo	5	2060	November 15, 2022
Loki error on port 9095 - error contacting scheduler Grafana Loki docker	3	7723	June 19, 2023
Loki doesn't work Grafana Loki loki	1	1645	March 28, 2024
Loki write pods not getting created but its showing running with 0/1 Grafana Loki loki	7	1352	February 25, 2025

Build cluster failed

Related topics