Hey,
I am trying to run loki in EKS using IPv6. After following a number of github issues (Loki unable to start on IPv6 EKS cluster · Issue #6251 · grafana/loki · GitHub in particular), I am able to start Loki up. However, when I try to write logs, I am getting the following error related to the ingester:
level=warn ts=2023-07-04T12:05:27.754911816Z caller=pool.go:193 msg="removing ingester failing healthcheck" addr=2600:1f11:908:b900:f342::a:9095 reason="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp: address 2600:1f11:908:b900:f342::a:9095: too many colons in address\""
level=warn ts=2023-07-04T12:05:27.754946634Z caller=pool.go:193 msg="removing ingester failing healthcheck" addr=2600:1f11:908:b901:de7::11:9095 reason="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp: address 2600:1f11:908:b901:de7::11:9095: too many colons in address\""
level=warn ts=2023-07-04T12:05:38.964732582Z caller=logging.go:86 traceID=2e07840a3b835c4a orgID=fake msg="POST /loki/api/v1/push (500) 3.529494ms Response: \"rpc error: code = Unavailable desc = connection error: desc = \\\"transport: Error while dialing dial tcp: address 2600:1f11:908:b901:de7::11:9095: too many colons in address\\\"\\n\" ws: false; Connection: close; Content-Length: 152470; Content-Type: application/x-protobuf; User-Agent: GrafanaAgent/; "
level=warn ts=2023-07-04T12:05:42.755648798Z caller=pool.go:193 msg="removing ingester failing healthcheck" addr=2600:1f11:908:b901:de7::11:9095 reason="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp: address 2600:1f11:908:b901:de7::11:9095: too many colons in address\""
level=warn ts=2023-07-04T12:05:42.755649919Z caller=pool.go:193 msg="removing ingester failing healthcheck" addr=2600:1f11:908:b900:f342::a:9095 reason="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp: address 2600:1f11:908:b900:f342::a:9095: too many colons in address\""
level=warn ts=2023-07-04T12:05:59.539257414Z caller=logging.go:86 traceID=5301968a412a8937 orgID=fake msg="POST /api/prom/push (500) 3.698641ms Response: \"rpc error: code = Unavailable desc = connection error: desc = \\\"transport: Error while dialing dial tcp: address 2600:1f11:908:b901:de7::11:9095: too many colons in address\\\"\\n\" ws: false; Content-Length: 168152; Content-Type: application/x-protobuf; User-Agent: promtail/; X-Scope-Orgid: \"\"; "
level=warn ts=2023-07-04T12:05:59.774633519Z caller=logging.go:86 traceID=3983451bd44bc58e orgID=fake msg="POST /api/prom/push (500) 31.088504ms Response: \"rpc error: code = Unavailable desc = connection error: desc = \\\"transport: Error while dialing dial tcp: address 2600:1f11:908:b901:de7::11:9095: too many colons in address\\\"\\n\" ws: false; Content-Length: 212772; Content-Type: application/x-protobuf; User-Agent: promtail/; X-Scope-Orgid: \"\"; "
level=warn ts=2023-07-04T12:06:00.698643077Z caller=logging.go:86 traceID=5f291bdf0cd34693 orgID=fake msg="POST /api/prom/push (500) 5.670154ms Response: \"rpc error: code = Unavailable desc = connection error: desc = \\\"transport: Error while dialing dial tcp: address 2600:1f11:908:b901:de7::11:9095: too many colons in address\\\"\\n\" ws: false; Content-Length: 212772; Content-Type: application/x-protobuf; User-Agent: promtail/; X-Scope-Orgid: \"\"; "
level=warn ts=2023-07-04T12:06:02.359378802Z caller=logging.go:86 traceID=6324c90323d18b4a orgID=fake msg="POST /api/prom/push (500) 4.75841ms Response: \"rpc error: code = Unavailable desc = connection error: desc = \\\"transport: Error while dialing dial tcp: address 2600:1f11:908:b901:de7::11:9095: too many colons in address\\\"\\n\" ws: false; Content-Length: 212772; Content-Type: application/x-protobuf; User-Agent: promtail/; X-Scope-Orgid: \"\"; "
level=warn ts=2023-07-04T12:06:03.705543294Z caller=logging.go:86 traceID=612a7729ca525377 orgID=fake msg="POST /loki/api/v1/push (500) 1.056221ms Response: \"rpc error: code = Unavailable desc = connection error: desc = \\\"transport: Error while dialing dial tcp: address 2600:1f11:908:b900:f342::a:9095: too many colons in address\\\"\\n\" ws: false; Connection: close; Content-Length: 31398; Content-Type: application/x-protobuf; User-Agent: GrafanaAgent/; "
level=warn ts=2023-07-04T12:06:06.095292975Z caller=logging.go:86 traceID=6539e2f39d446d32 orgID=fake msg="POST /api/prom/push (500) 10.102688ms Response: \"rpc error: code = Unavailable desc = connection error: desc = \\\"transport: Error while dialing dial tcp: address 2600:1f11:908:b901:de7::11:9095: too many colons in address\\\"\\n\" ws: false; Content-Length: 212772; Content-Type: application/x-protobuf; User-Agent: promtail/; X-Scope-Orgid: \"\"; "
level=warn ts=2023-07-04T12:06:12.471044607Z caller=logging.go:86 traceID=6437e80eac365c40 orgID=fake msg="POST /api/prom/push (500) 5.533771ms Response: \"rpc error: code = Unavailable desc = connection error: desc = \\\"transport: Error while dialing dial tcp: address 2600:1f11:908:b900:f342::a:9095: too many colons in address\\\"\\n\" ws: false; Content-Length: 212772; Content-Type: application/x-protobuf; User-Agent: promtail/; X-Scope-Orgid: \"\"; "
level=info ts=2023-07-04T12:06:12.743950433Z caller=table_manager.go:134 msg="uploading tables"
level=info ts=2023-07-04T12:06:12.746168582Z caller=table_manager.go:166 msg="handing over indexes to shipper"
level=warn ts=2023-07-04T12:06:12.754793371Z caller=pool.go:193 msg="removing ingester failing healthcheck" addr=2600:1f11:908:b900:f342::a:9095 reason="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp: address 2600:1f11:908:b900:f342::a:9095: too many colons in address\""
level=warn ts=2023-07-04T12:06:12.754813794Z caller=pool.go:193 msg="removing ingester failing healthcheck" addr=2600:1f11:908:b901:de7::11:9095 reason="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp: address 2600:1f11:908:b901:de7::11:9095: too many colons in address\""
I have also tried adding brackets around the IPv6 pod IP as outlined here. simple scalable deployment: rpc error: code = Unimplemented desc = unknown service logproto.Querier · Issue #5578 · grafana/loki · GitHub, but when I do that the pods fail to start.
Has anyone seen this behaviour?
Thanks