Tribe node binding to incorrect interface after upgrade to 1.5.2

[nemonster]

Current configuration consists of 2 tribe nodes connecting to 3 clusters, with each cluster containing 3 data nodes and a dedicated client instance. One tribe node connects to clusters 1-2, and one
connects to clusters 1-3.

All the client nodes and Tribe nodes run on the same server.. This server has 2 network
interfaces, one is a public interface, the other is on a private interface on bond0. Each Tribe node and Client node provide http connection via the public interface and and transport connection via the bond0 interface. This setup works fine for both Tribe and Client in 1.4.2 but only works
for the Client node in 1.5.2. The Tribe node, even when using the exact same elasticsearch.yml file as the 1.4.2 version, wants to bind to the public interface for the both the http and transport when starting up using the 1.5.2 version.

Log comparison (actual values changed)

Starts out:
INFO ][node ] [abc-es-tribe2-t01] starting ...
INFO ][transport ] [abc-es-tribe2-t01] bound_address {inet[/192.168.52.62:8550]}, publish_address {inet[/192.168.52.62:8550]}

8 lines later:

[2015-06-02 09:05:22,242][INFO ][node ] [abc-es-tribe2-t01/routing] starting ...
[2015-06-02 09:05:22,314][INFO ][transport ] [abc-es-tribe2-t01/routing] bound_address {inet[/0.0.0.0:9301]}, publish_address {inet[/xxx.xx.12.82:9301]}

The Tribe is coming up on the correct interfaces per the elasticsearch.yml file settings. But, when the routing starts, it binds the transport to the wrong interface, resulting in:

INFO ][discovery.zen ] [abc-es-tribe2-t01/t1] failed to send join request to master ...internal:discovery/zen/join]];
nested: ConnectTransportException[[abc-es-tribe2-t01/t1][inet[/xxx.xx.12.82:9302]] connect_timeout[2m]]; nested: SocketException[Network is unreachable]; ]

network:
bind_host: bond0
publish_host: bond0

transport:
host: 192.168.52.62:8550
tcp:
port: 8550-8999

[clintongormley]

Hi @nemonster

This is probably due to this bug fix: #9721. Probably the tribe was relying on settings that are now excluded from the tribe client settings. Can you upload the config files?

/cc @javanna

[javanna]

Yes sounds like #9721 might be the cause, I think it can be solved by adding the needed settings to the tribes configuration specifically, rather than relying on global settings being forwarded to the tribe automatically. Let's have a look at the config file to find out which settings this could be.

[dhramveersingh]

Hi, I am facing the similar problem. details as below:

Software versions
a) Elasticsearch 1.6.0
b) Logstash 1.5.1
c) Kibana 4.1.0-linux-x64

Configuration of tribe node:
"-Des.http.port=29200"
"-Des.transport.tcp.port=29300-29349"
"-Des.network.publish_host=localhost"
"-Des.node.name=global.node1"
"-Des.tribe.hk.cluster.name=dev.hk"
"-Des.tribe.uk.cluster.name=dev.uk"
"-Des.discovery.zen.ping.unicast.hosts=localhost:29350,localhost:29351"
"-Des.node.data=false"
"-Des.node.master=false"

Configuration of node UK:
"-Des.http.port=29250"
"-Des.transport.tcp.port=29350"
"-Des.network.publish_host=localhost"
"-Des.discovery.zen.ping.unicast.hosts=localhost:29200"
"-Des.discovery.zen.ping.multicast.enabled=false"

Configuration of node HK:
"-Des.http.port=29251"
"-Des.transport.tcp.port=29351"
"-Des.network.publish_host=localhost"
"-Des.discovery.zen.ping.unicast.hosts=localhost:29200"
"-Des.discovery.zen.ping.multicast.enabled=false"

Issue
log of the tribe, where it is clearly visible that, global.node1/uk and global.node1/hk are trying to connect to the 9300 and 9301 port instead of the 29300 and 29301 ports as specified in the configuration files.

[2015-07-20 17:06:50,872][INFO ][node ] [global.node1] starting ...
[2015-07-20 17:06:50,973][INFO ][transport ] [global.node1] bound_address {inet[/0.0.0.0:29300]}, publish_address {inet[localhost/10.117.20.24:29300]}
[2015-07-20 17:06:50,984][INFO ][discovery ] [global.node1] elasticsearch/bUO32MNNScSAjZYHNwJIlw
[2015-07-20 17:06:50,984][WARN ][discovery ] [global.node1] waited for 0s and no initial state was set by the discovery
[2015-07-20 17:06:51,002][INFO ][http ] [global.node1] bound_address {inet[/0.0.0.0:29200]}, publish_address {inet[localhost/10.117.20.24:29200]}
[2015-07-20 17:06:51,002][INFO ][node ] [global.node1/hk] starting ...
[2015-07-20 17:06:51,046][INFO ][transport ] [global.node1/hk] bound_address {inet[/0.0.0.0:9300]}, publish_address {inet[/169.254.95.119:9300]}
[2015-07-20 17:06:51,069][INFO ][discovery ] [global.node1/hk] es.hk/Ja3FpoMXTKqMs4_q7AhIpw

[clintongormley]

This has been fixed in recent releases. Closing