-
Notifications
You must be signed in to change notification settings - Fork 839
Description
Is your feature request related to a problem? Please describe.
I am trying to use memberlist to connect cortex microservices in docker across host (via nomad) with consul DNS. They can connect to each other fine using
memberlist:
node_name: {{ env "NOMAD_JOB_NAME" }}-{{ env "NOMAD_TASK_NAME" }}-{{ env "NOMAD_ALLOC_INDEX" }}
# https://cortexmetrics.io/docs/configuration/arguments/#dns-service-discovery
join_members:
# - dnssrv+tempo-ingester.service.consul
- dnssrv+memberlist.tempo-distributor.service.consul
# - dnssrv+tempo-querier.service.consulbecause this DNS resolves to a 10.0.*.* internal IP. However, when they try to write to the KV store I see
level=warn ts=2021-07-12T05:55:46.980062631Z caller=tcp_transport.go:417 msg="TCPTransport: WriteTo failed" addr=172.26.64.20:26979 err="dial tcp 172.26.64.20:26979: connect: no route to host"
level=warn ts=2021-07-12T05:55:50.052034827Z caller=tcp_transport.go:417 msg="TCPTransport: WriteTo failed" addr=172.26.64.20:26979 err="dial tcp 172.26.64.20:26979: connect: no route to host"
level=warn ts=2021-07-12T05:55:57.764027018Z caller=tcp_transport.go:417 msg="TCPTransport: WriteTo failed" addr=172.26.64.20:26979 err="dial tcp 172.26.64.20:26979: connect: no route to host"
level=warn ts=2021-07-12T05:56:00.836011795Z caller=tcp_transport.go:417 msg="TCPTransport: WriteTo failed" addr=172.26.64.20:26979 err="dial tcp 172.26.64.20:26979: connect: no route to host"
level=warn ts=2021-07-12T05:56:07.779901566Z caller=tcp_transport.go:417 msg="TCPTransport: WriteTo failed" addr=172.26.64.20:26979 err="dial tcp 172.26.64.20:26979: connect: no route to host"
because they're on different hosts and cannot communicate over the 172.16.*.* network. It seems like memberlist is using the IP of the internal docker interface and can be overridden with AdvertiseAddr, but I don't see any way to specify it in the cortex configuration.
Describe the solution you'd like
Add support for AdvertiseAddr and AdvertisePort in the memberlist configuration. Happy to submit a PR, but wanted to make sure I wasn't missing anything or doing something incorrectly!