Good morning all!
I’m struggling with Upgrading my loki installation from 2.7 to 2.9. We are using loki-distributed helm chart to deploy loki on our k8s cluster. When trying to install the latest version I get troubles starting up the memberlist service.
I’ve tried various configuration options, but all do fail. With loki 2.7 I used this configuration and it worked:
extraEnv:
-
- name: MY_POD_IP*
- valueFrom:*
-
fieldRef:* -
fieldPath: status.podIP*
extraArgs:
-
- -config.expand-env=true*
-
- -log-config-reverse-order*
-
- -memberlist.bind-addr=$(MY_POD_IP)*
Now with loki 2.9 I’m getting this error message:
level=error ts=2023-10-04T07:05:33.696841705Z caller=loki.go:517 msg=“module failed” module=memberlist-kv error=“invalid service state: Failed, expected: Running, failure: failed to create transport: failed to start TCP listener on \“11.32.28.15\” port 7946: listen tcp 11.32.28.15:7946: bind: address already in use”
Removing the memberlist.bind-addr argument fails with also:
level=error ts=2023-10-04T07:40:27.632708554Z caller=loki.go:517 msg=“module failed” module=memberlist-kv error=“invalid service state: Failed, expected: Running, failure: failed to create memberlist: Failed to get final advertise address: no private IP address found, and explicit IP not provided”
Update:
The bind address already in use error is probably thrown because in the generated config the same IP configured twice:
level=info ts=2023-10-04T08:00:08.229376351Z caller=config.go:26 type=config msg=” - 11.32.28.15"
level=info ts=2023-10-04T08:00:08.229379215Z caller=config.go:26 type=config msg=” - 11.32.28.15"
level=info ts=2023-10-04T08:00:08.229382005Z caller=config.go:26 type=config msg=” bind_addr:”
This sounds like this loki issue: Using flag -memberlist.bind-addr results in duplicates in config · Issue #2363 · grafana/loki · GitHub (not solved yet?)
Has anyone a solution for this issue? Thanks for help!