Make `node.master` a dynamic setting

[TwP]

It would be nice to add dedicated master nodes to an existing cluster without requiring a full restart of each node in the cluster. To accomplish this, the node.master would need to be dynamically configurable.

Changing the node.master setting at runtime would not force a new master election. It would only apply to the cluster moving forward. To force a master election, the current master node would need to be restarted as is currently the case today.

[s1monw]

@TwP we will discuss this internally but it might take a while to get here, just a headsup

[clintongormley]

I like the idea. That said:

It would be nice to add dedicated master nodes to an existing cluster without requiring a full restart of each node in the cluster.

You currently only need to restart the node you're promoting to master, unless I misunderstand you?

[dakrone]

You currently only need to restart the node you're promoting to master, unless I misunderstand you?

Let's say you start with a 9-node cluster with minimum_master_nodes set to 5, then you decide at some point you want to move to dedicated master nodes. If the node.master setting were dynamic, you could add 3 nodes with node.master: true and update node.master: false on the other 9 nodes (also updating minimum_master_nodes to 2). Then you could (optionally) bounce the current master node and have one of the dedicated master nodes take over.

Right now you would have to restart each of the 9 data nodes (which stinks if you have a lot of data) in order to mark them all as non-master-eligible, because the node.master setting can't be dynamically changed.

[clintongormley]

OK. I can see one issue here. If you change the setting dynamically and you don't update the config file, and the node reboots (expectedly or unexpectedly), it'll pick up the node.master setting from the config file.

Currently, all node-specific (as opposed to cluster-wide) settings are set at node startup only. We have no API to set node settings, and nowhere to persist them.

[clintongormley]

Isn't the better solution for this problem fixing the slow restarts?

[dakrone]

Isn't the better solution for this problem fixing the slow restarts?

I don't think so, (don't get me wrong, fixing slow restarts would be fantastic), but I still think it is nice to be able to make changes to node.master without requiring a restart.

[rjernst]

A real world scenario where you may want this is when transitioning to a new set of nodes in a cloud environment. You would want to nicely transition to those new nodes, but you need to force the system to transition off of the old nodes. While killing the old nodes should work, this makes the exceptional case (ie master dies) the path taken, which shouldn't be necessary. There should be a clean way to transition from one set of master nodes to another set.

[s1monw]

This kind of scenario is something that wonders me for a while now. You have a setup where you realize it`s not optimal ie. want to move toe dedicated master nodes etc. But you have to change this node level setting and bounce processes, do recoveries etc. set minimum master nodes what have you. (I bet folks miss at least on important step here regularly). I wonder if we should have a dedicated API that bakes the cluster. Ie you wanna move to 3 master nodes from a 9 node cluster you would have ideally a simple API call like this:

PUT /_cluster/bake
{
  "master_nodes" : [ "master_1", "master_2", "master_3"],
  "set_minimum_master_nodes" : "true|false #optional true by default", 
  "force" : "true|false #false by default to barf if something is not safe (ie. only one master)"
}

with this call we basically move away form everything is dynamic and ignore other master nodes even if they are master eligible. It will force a new election if the current master is not in the list. This might also help use with some safety mechanisms or allows us to harden master election since we know the nodes now and the list is not dynamic anymore. I am not an expert on this but I wanted to throw out the idea here...

[clintongormley]

You still have to deal with what happens if any nodes reboot unexpectedly. Do they take their local elasticsearch.yml into account, or just use whatever setting is current in the cluster? If the latter, then the settings applied via the API should always override the local settings (which is confusing if you see settings from the yaml not being applied). What happens if three ex-masters reboot, and can't see the rest of the cluster initially? They'd use their local settings and form their own cluster.

[TwP]

@clintongormley you bring up a very valid point. However, that problem is not isolated to this proposed change to the node.master eligibility setting. All dynamically configurable settings have this same problem. Although with the node.master eligibility the consequences could be much more severe as you pointed out.

The first step is making the setting dynamically configurable. Changing the master eligibility via the API is one route for setting this value. The other route is to update the elasticsearch.yaml configuration file and then signal the running process to reload settings from the configuration file. Static settings are ignored. Dynamic settings are updated accordingly.

How dynamic settings are handled across restarts is an orthogonal problem. It should not prevent this feature from being implemented. However, taking the broader view will definitely highlight problems that can be introduced by making master eligibility configurable at runtime.

[mlorch-ai]

I just ran into the situation of having to restart a full production cluster to enable this setting and hence would've loved it we could change that dynamically.
Please keep in mind that if we change the master node (e.g. going from a cluster where each node can be a master to a cluster with a smaller number of dedicated masters) we also need to change/adapt other settings (such as minimum_master_nodes) dynamically.

I dont see an issue with the fact that the config file may have outdated information. We need to keep the config file updated with other settings as well as we evolve our cluster to accomodate for node failures/restarts. Being able to change this dynamically would however avoid the "yellow" cluster state and reduced redundancy that happens when a node (all nodes) needs to be restarted while indexing is going on as its indexes need to be brought up to speed.

[rjsm]

I would assume any cluster that this would be an issue on is under config management. You can update the elasticsearch.yml under the node, and then tell to not be master eligible. If it unexpectedly restarts, it'll pick up the intended configuration from the file. I'm transitioning my cluster to dedicated masters here shortly.