Skip to content

add advertise_addr and advertise_port to memberlist configuration #4359

@kevinschoonover

Description

@kevinschoonover

Is your feature request related to a problem? Please describe.
I am trying to use memberlist to connect cortex microservices in docker across host (via nomad) with consul DNS. They can connect to each other fine using

memberlist:
  node_name: {{ env "NOMAD_JOB_NAME" }}-{{ env "NOMAD_TASK_NAME" }}-{{ env "NOMAD_ALLOC_INDEX" }}
  # https://cortexmetrics.io/docs/configuration/arguments/#dns-service-discovery
  join_members:
    # - dnssrv+tempo-ingester.service.consul
    - dnssrv+memberlist.tempo-distributor.service.consul
    # - dnssrv+tempo-querier.service.consul

because this DNS resolves to a 10.0.*.* internal IP. However, when they try to write to the KV store I see

level=warn ts=2021-07-12T05:55:46.980062631Z caller=tcp_transport.go:417 msg="TCPTransport: WriteTo failed" addr=172.26.64.20:26979 err="dial tcp 172.26.64.20:26979: connect: no route to host"
level=warn ts=2021-07-12T05:55:50.052034827Z caller=tcp_transport.go:417 msg="TCPTransport: WriteTo failed" addr=172.26.64.20:26979 err="dial tcp 172.26.64.20:26979: connect: no route to host"
level=warn ts=2021-07-12T05:55:57.764027018Z caller=tcp_transport.go:417 msg="TCPTransport: WriteTo failed" addr=172.26.64.20:26979 err="dial tcp 172.26.64.20:26979: connect: no route to host"
level=warn ts=2021-07-12T05:56:00.836011795Z caller=tcp_transport.go:417 msg="TCPTransport: WriteTo failed" addr=172.26.64.20:26979 err="dial tcp 172.26.64.20:26979: connect: no route to host"
level=warn ts=2021-07-12T05:56:07.779901566Z caller=tcp_transport.go:417 msg="TCPTransport: WriteTo failed" addr=172.26.64.20:26979 err="dial tcp 172.26.64.20:26979: connect: no route to host"

because they're on different hosts and cannot communicate over the 172.16.*.* network. It seems like memberlist is using the IP of the internal docker interface and can be overridden with AdvertiseAddr, but I don't see any way to specify it in the cortex configuration.

Describe the solution you'd like
Add support for AdvertiseAddr and AdvertisePort in the memberlist configuration. Happy to submit a PR, but wanted to make sure I wasn't missing anything or doing something incorrectly!

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions