Voting Only Master Node

[pickypg]

The current ideal setup for master nodes is to have 3 master nodes, thus giving high availability (HA) with a quorum of 2 master nodes at any given time.

However, in some cloud regions (and more frequently in internal data centers), this creates a problem because there may only be two available zones to place your nodes. In these scenarios, you do not want to give up HA just to maintain 3 master nodes (to be clear: it's better than not doing it, but it's less than ideal) because it means that you have a lopsided HA environment where one "half" cannot actually survive without the other.

The only real solution to this problem is to have a third zone, even if that third zone is a slightly higher latency (across the globe is not going to work), then plunk a master node into that zone to give a single master node per zone -- thus enabling the actual survivability of any single zone, but not of two zones.

Now, you could do this today with a master node and things would be generally okay except that the master node with the added latency may become the elected master node, which is obviously not ideal.

This is where a voting-only master node improves this setup significantly. If you have a master node that can itself not be elected, but that participates in elections only, then you can have a tiny node (careful with VM size due to automatic network throttling in cloud ecosystems!) that help to survive the loss of any single zone without seriously risking performance.

[bleskes]

This is definitely on our horizon. We need to do some infra work before we get there though... on it :)

[bertrandfalguiere]

I'm new to ES, but won't it be better and maybe technically easier to give "points" to the master eligible nodes? Those points would give priority to the best ranked node to become master.
If zones A and B have high availability, but not zone C, you could give:
"node.master_eligibility" : 3 for your master eligible node A,
"node.master_eligibility" : 2 for your master eligible node B,
"node.master_eligibility" : 1 for your laggy master eligible node C.
If you lose A, B is elected if needed, and vice versa.
It would have have the same effect of avoiding C to become master.

And (new case) if you have a fast A and slow B and C, it means that B and C can take over if the cluster lost A, but as soon as the fast A node is back online, it takes its mastership back.

It probably means changing the election process, but no new node status is needed.

Does it make sense? Again i'm new to all this.

[bleskes]

@bertrandfalguiere no worries and welcome to the repo.

If I read you correctly, you want to have some mechanics to influence master election and indicate you prefer one master over another. Putting all the trickiness of a distributed master election aside (it's not easy to add these things correctly), the main thing to realize here that you need to have a good working master at any given moment. Otherwise your cluster is in a big risk. So have a slow master and a fast master doesn't make sense - you need two fast masters etc.

PS. We try to keep github for concrete issues. For more discussion-y things we ask people to go to https://discuss.elastic.co

[elasticmachine]

Pinging @elastic/es-distributed

[seang-es]

In some cloud use cases we might have a two zone deployment with a third zone containing only a tiebreaker master node. By default, this will be configured with 1 GB RAM, and may not be suitable for larger clusters. If this could be configured as a voting-only node, then it could provide quorum without becoming an active master.