Gossip-ring / memberlist - no private IP address found

Grafana Tempo
8

We had this problem when our pod’s IP range was 240.x.x.x, which is reserved, and got filtered out (Note: there are other ranges that are also filtered out go-sockaddr/rfc.go at 6d291a969b86c4b633730bfc6b8b9d64c3aafed9 · hashicorp/go-sockaddr · GitHub).

Changing the pod’s IP range was not an option, so to solve this, we had to set the memberlist bind address in the config file explicitly to the pod’s ip address.

To do this, you need to:

  1. create an environment variable with the pod’s ip (in values.yaml):
ingester:
  extraEnv:
    - name: MY_POD_IP
      valueFrom:
        fieldRef:
          fieldPath: status.podIP
  1. Allow environment variable expansion in the config file (do this per service in values.yaml):
ingester:
  extraArgs:
    - -config.expand-env=true
  1. Specify the memberlist bind address for the config file (also in values.yaml):
config;
  memberlist:
    bind_addr:
      - ${MY_POD_IP}
4 Likes