unusable hostnames broadcast for inter-node communication

[sploiselle]

When starting cockroach without the --host flag, Cockroach chooses the machine's host name, which it broadcasts as the address other nodes in the cluster can use to communicate to it. However, this name isn't necessarily accessible and the only option to change it is to use the --host flag, which limits the number of interfaces Cockroach listens on.

Example

If I called my machine do-node-2 and start cockroach without the --host flag, the node broadcasts itself as do-node-2:26257. Often times, that name doesn't have requisite DNS entries and is accessible only when the machine refers to itself.

Again, resolving this behavior --host is not viable in all scenarios because the behavior changes from listening on all interfaces to listening only on a single one.

Consequence

• Having nodes join clusters is difficult because it requires setting the --host flag
• Nodes cannot replicate its data to others in the cluster if they use addresses like this, despite the fact that they all joined the cluster successfully. Attached is an example of this happening on Digital Ocean with 3 droplets. The first node started the cluster using the following command:

cockroach start --insecure --background --host=10.132.80.187

The second and third nodes joined the cluster using this command:

cockroach start --insecure --background --join=10.132.80.187:26257

Ranges on the first node cannot be replicated to the second and third nodes because the addresses they're broadcasting are unreachable by the first node (i.e., they don't have DNS entries to point to do-node-2 and 3).

[tamird]

Dupe of #1008?

[sploiselle]

@tamird: Marc asked for a second ticket because of the potential one-way replication issue

[tamird]

OK. cc @a-robinson to avoid duplication of effort.

[mberhault]

it's a bit more than that. In this case, nodes 2 and 3 joined the cluster properly because they were able to reach node1. However, node 1 could not initiate a connection to the other two, meaning we were stuck with a single-node cluster.
The admin UI showed no sign of this. So while #1008 would solve the addressing issue, we still need to figure out a way to properly surface this scenario.

This would be needed for other uni-directional cases, not just bad advertised address. eg: a badly configured firewall rule will trivially cause this.

[petermattis]

@tamird, @mberhault We now have the --advertise-host flag. Is there anything left to do here?

[tamird]

@petermattis @mberhault the --advertise-host flag was added in #9503, which also closed #1008, but this issue is (according to @mberhault) not a duplicate of #1008.

@mberhault @sploiselle can you clarify?

[a-robinson]

It seems like there's still the matter of helping folks who accidentally run into this. There's no real visibility into what's wrong in cases like this.

[sploiselle]

This is a UX issue, as @a-robinson said. Nodes join the cluster but will never successfully receive data. Identifying this could be non-obvious to users because everything appears to be working from the CLI.

[petermattis]

Perhaps the new cluster visualization could highlight this situation. Cc @maxlang, @mrtracy, @kuanluo.

[spencerkimball]

Does anyone have a concrete suggestion for how to address this? I'm removing the 1.0 milestone because I wasn't able to come up with a decent idea. Note that there are complaints in the logs about this situation:

I170402 16:25:37.422167 236 vendor/google.golang.org/grpc/clientconn.go:806  grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp 198.105.244.228:26259: i/o timeout"; Reconnecting to {f34tegaf.com:26259 <nil>}

[petermattis]

Might be fixed by #16177.

[tbg]

This particular one is indeed fixed by #16177, except in the case where the local host can resolve the hostname. I think what's left here is some general warning mechanism that fires when other nodes can't reach out at the advertised address, but it works the other way. That is tracked in #18850.