Skip to content

Unable to update discovery.zen.minimum_master_nodes #10614

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
tomas-edwardsson opened this issue Apr 15, 2015 · 8 comments
Closed

Unable to update discovery.zen.minimum_master_nodes #10614

tomas-edwardsson opened this issue Apr 15, 2015 · 8 comments
Assignees
@dadoonet
Labels
>bug

Comments

@tomas-edwardsson
Copy link

I've been trying to update discovery.zen.minimum_master_nodes on my elasticsearch cluster. It seems that nothing gets updated and nothing logged.

Running version 1.5.1

I also tried updating only transient which behaved the same way.

curl -XPUT localhost:9200/_cluster/settings -d '{
    "persistent" : {
        "discovery.zen.minimum_master_nodes" : 3
    },
    "transient" : {
        "discovery.zen.minimum_master_nodes" : 3
    }
}'
{"acknowledged":true,"persistent":{},"transient":{}}

Checking the settings afterwards I get:

curl -s http://localhost:9200/_cluster/settings?pretty=true
{
  "persistent" : {
    "discovery" : {
      "zen" : {
        "minimum_master_nodes" : "1"
      }
    }
  },
  "transient" : {
    "cluster" : {
      "routing" : {
        "allocation" : {
          "enable" : "all"
        }
      }
    },
    "discovery" : {
      "zen" : {
        "minimum_master_nodes" : "1"
      }
    }
  }
}
@imotov
Copy link
Contributor

imotov commented Apr 16, 2015

@tomas-edwardsson how many master eligible nodes do you have in the cluster?

@tomas-edwardsson
Copy link
Author

I have 5 nodes, 3 with node.data: true and 2 with node.data: false

@imotov
Copy link
Contributor

imotov commented Apr 17, 2015

If it means that you have only 2 nodes with node.master:true setting minimum master nodes to 3 would make your cluster to inoperational until the third node with node.master : true is added. So elasticsearch prevents this situation by ignoring the setting. Check log on master. There should be a warning there.

@tomas-edwardsson
Copy link
Author

All the nodes have node.master: true.

@imotov
Copy link
Contributor

imotov commented Apr 21, 2015

@tomas-edwardsson that's strange, could you post the output of the following command here

curl "localhost:9200/_nodes/settings?pretty"

@tomas-edwardsson
Copy link
Author 

{
  "cluster_name" : "app",
  "nodes" : {
    "cwFOF14ETsKldIRdQ_L3Yg" : {
      "name" : "app/node01",
      "transport_address" : "inet[/192.168.1.1:9300]",
      "host" : "app-1",
      "ip" : "127.0.0.1",
      "version" : "1.5.1",
      "build" : "5e38401",
      "http_address" : "inet[/192.168.1.1:9200]",
      "attributes" : {
        "aws_availability_zone" : "us-west-1a",
        "master" : "true"
      },
      "settings" : {
        "index" : {
          "number_of_replicas" : "1",
          "number_of_shards" : "10",
          "search" : {
            "slowlog" : {
              "threshold" : {
                "fetch" : {
                  "warn" : "1s",
                  "debug" : "500ms",
                  "trace" : "200ms",
                  "info" : "800ms"
                }
              }
            }
          },
          "refresh_interval" : "30"
        },
        "bootstrap" : {
          "mlockall" : "true"
        },
        "client" : {
          "type" : "node"
        },
        "script" : {
          "disable_dynamic" : "true"
        },
        "plugin" : {
          "mandatory" : "cloud-aws"
        },
        "node" : {
          "data" : "true",
          "master" : "true",
          "name" : "app/node01"
        },
        "http" : {
          "cors" : {
            "enabled" : "true",
            "allow-origin" : "/https?:\\/\\/.*\\.example\\.com/"
          }
        },
        "name" : "app/node01",
        "path" : {
          "data" : "/var/lib/elasticsearch/data",
          "work" : "/var/lib/elasticsearch",
          "home" : "/usr/share/elasticsearch",
          "conf" : "/etc/elasticsearch",
          "logs" : "/var/log/elasticsearch"
        },
        "cloud" : {
          "aws" : {
            "region" : "us-west-1"
          },
          "node" : {
            "auto_attributes" : "true"
          }
        },
        "config" : "/etc/elasticsearch/elasticsearch.yml",
        "cluster" : {
          "name" : "app"
        },
        "indices" : {
          "cache" : {
            "filter" : {
              "size" : "30%"
            }
          },
          "memory" : {
            "index_buffer_size" : "50%"
          }
        },
        "discovery" : {
          "type" : "ec2",
          "ec2" : {
            "groups" : "appfeed"
          }
        },
        "foreground" : "yes"
      }
    },
    "Pg3-SmS9RrqKbia2fPU13A" : {
      "name" : "feeder/node01",
      "transport_address" : "inet[/192.168.1.2:9300]",
      "host" : "feeder-1",
      "ip" : "127.0.0.1",
      "version" : "1.5.1",
      "build" : "5e38401",
      "http_address" : "inet[/192.168.1.2:9200]",
      "attributes" : {
        "aws_availability_zone" : "us-west-1a",
        "data" : "false",
        "master" : "true"
      },
      "settings" : {
        "index" : {
          "number_of_replicas" : "1",
          "number_of_shards" : "10",
          "search" : {
            "slowlog" : {
              "threshold" : {
                "fetch" : {
                  "warn" : "1s",
                  "debug" : "500ms",
                  "trace" : "200ms",
                  "info" : "800ms"
                }
              }
            }
          },
          "refresh_interval" : "30"
        },
        "bootstrap" : {
          "mlockall" : "true"
        },
        "client" : {
          "type" : "node"
        },
        "script" : {
          "disable_dynamic" : "true"
        },
        "plugin" : {
          "mandatory" : "cloud-aws"
        },
        "node" : {
          "data" : "false",
          "master" : "true",
          "name" : "feeder/node01"
        },
        "http" : {
          "cors" : {
            "enabled" : "true",
            "allow-origin" : "/https?:\\/\\/.*\\.example\\.com/"
          }
        },
        "name" : "feeder/node01",
        "path" : {
          "data" : "/var/lib/elasticsearch/data",
          "work" : "/var/lib/elasticsearch",
          "home" : "/usr/share/elasticsearch",
          "conf" : "/etc/elasticsearch",
          "logs" : "/var/log/elasticsearch"
        },
        "cloud" : {
          "aws" : {
            "region" : "us-west-1"
          },
          "node" : {
            "auto_attributes" : "true"
          }
        },
        "config" : "/etc/elasticsearch/elasticsearch.yml",
        "cluster" : {
          "name" : "app"
        },
        "indices" : {
          "cache" : {
            "filter" : {
              "size" : "30%"
            }
          },
          "memory" : {
            "index_buffer_size" : "50%"
          }
        },
        "discovery" : {
          "type" : "ec2",
          "ec2" : {
            "groups" : "appfeed"
          }
        },
        "foreground" : "yes"
      }
    },
    "UyCcs5A1Q-mHdy5AubnW5w" : {
      "name" : "app/node03",
      "transport_address" : "inet[/192.168.1.3:9300]",
      "host" : "app-3",
      "ip" : "127.0.0.1",
      "version" : "1.5.1",
      "build" : "5e38401",
      "http_address" : "inet[/192.168.1.3:9200]",
      "attributes" : {
        "aws_availability_zone" : "us-west-1c",
        "master" : "true"
      },
      "settings" : {
        "index" : {
          "number_of_replicas" : "1",
          "number_of_shards" : "10",
          "search" : {
            "slowlog" : {
              "threshold" : {
                "fetch" : {
                  "warn" : "1s",
                  "debug" : "500ms",
                  "trace" : "200ms",
                  "info" : "800ms"
                }
              }
            }
          },
          "refresh_interval" : "30"
        },
        "bootstrap" : {
          "mlockall" : "true"
        },
        "client" : {
          "type" : "node"
        },
        "script" : {
          "disable_dynamic" : "true"
        },
        "plugin" : {
          "mandatory" : "cloud-aws"
        },
        "node" : {
          "data" : "true",
          "master" : "true",
          "name" : "app/node03"
        },
        "http" : {
          "cors" : {
            "enabled" : "true",
            "allow-origin" : "/https?:\\/\\/.*\\.example\\.com/"
          }
        },
        "name" : "app/node03",
        "path" : {
          "data" : "/var/lib/elasticsearch/data",
          "work" : "/var/lib/elasticsearch",
          "home" : "/usr/share/elasticsearch",
          "conf" : "/etc/elasticsearch",
          "logs" : "/var/log/elasticsearch"
        },
        "cloud" : {
          "aws" : {
            "region" : "us-west-1"
          },
          "node" : {
            "auto_attributes" : "true"
          }
        },
        "config" : "/etc/elasticsearch/elasticsearch.yml",
        "cluster" : {
          "name" : "app"
        },
        "indices" : {
          "cache" : {
            "filter" : {
              "size" : "30%"
            }
          },
          "memory" : {
            "index_buffer_size" : "50%"
          }
        },
        "discovery" : {
          "type" : "ec2",
          "ec2" : {
            "groups" : "appfeed"
          }
        },
        "foreground" : "yes"
      }
    },
    "LOyQBO60Q8qMfWqEksCAbQ" : {
      "name" : "feeder/node02",
      "transport_address" : "inet[/192.168.1.4:9300]",
      "host" : "feeder-2",
      "ip" : "127.0.0.1",
      "version" : "1.5.1",
      "build" : "5e38401",
      "http_address" : "inet[/192.168.1.4:9200]",
      "attributes" : {
        "aws_availability_zone" : "us-west-1b",
        "data" : "false",
        "master" : "true"
      },
      "settings" : {
        "index" : {
          "number_of_replicas" : "1",
          "number_of_shards" : "10",
          "search" : {
            "slowlog" : {
              "threshold" : {
                "fetch" : {
                  "warn" : "1s",
                  "debug" : "500ms",
                  "trace" : "200ms",
                  "info" : "800ms"
                }
              }
            }
          },
          "refresh_interval" : "30"
        },
        "bootstrap" : {
          "mlockall" : "true"
        },
        "client" : {
          "type" : "node"
        },
        "script" : {
          "disable_dynamic" : "true"
        },
        "plugin" : {
          "mandatory" : "cloud-aws"
        },
        "node" : {
          "data" : "false",
          "master" : "true",
          "name" : "feeder/node02"
        },
        "http" : {
          "cors" : {
            "enabled" : "true",
            "allow-origin" : "/https?:\\/\\/.*\\.example\\.com/"
          }
        },
        "name" : "feeder/node02",
        "path" : {
          "data" : "/var/lib/elasticsearch/data",
          "work" : "/var/lib/elasticsearch",
          "home" : "/usr/share/elasticsearch",
          "conf" : "/etc/elasticsearch",
          "logs" : "/var/log/elasticsearch"
        },
        "cloud" : {
          "aws" : {
            "region" : "us-west-1"
          },
          "node" : {
            "auto_attributes" : "true"
          }
        },
        "config" : "/etc/elasticsearch/elasticsearch.yml",
        "cluster" : {
          "name" : "app"
        },
        "indices" : {
          "cache" : {
            "filter" : {
              "size" : "30%"
            }
          },
          "memory" : {
            "index_buffer_size" : "50%"
          }
        },
        "discovery" : {
          "type" : "ec2",
          "ec2" : {
            "groups" : "appfeed"
          }
        },
        "foreground" : "yes"
      }
    },
    "m-36w4hyQN24gMH1lRsLPA" : {
      "name" : "app/node02",
      "transport_address" : "inet[/192.168.1.5:9300]",
      "host" : "app-2",
      "ip" : "127.0.0.1",
      "version" : "1.5.1",
      "build" : "5e38401",
      "http_address" : "inet[/192.168.1.5:9200]",
      "attributes" : {
        "aws_availability_zone" : "us-west-1b",
        "master" : "true"
      },
      "settings" : {
        "index" : {
          "number_of_replicas" : "1",
          "number_of_shards" : "10",
          "search" : {
            "slowlog" : {
              "threshold" : {
                "fetch" : {
                  "warn" : "1s",
                  "debug" : "500ms",
                  "trace" : "200ms",
                  "info" : "800ms"
                }
              }
            }
          },
          "refresh_interval" : "30"
        },
        "bootstrap" : {
          "mlockall" : "true"
        },
        "client" : {
          "type" : "node"
        },
        "script" : {
          "disable_dynamic" : "true"
        },
        "plugin" : {
          "mandatory" : "cloud-aws"
        },
        "node" : {
          "data" : "true",
          "master" : "true",
          "name" : "app/node02"
        },
        "http" : {
          "cors" : {
            "enabled" : "true",
            "allow-origin" : "/https?:\\/\\/.*\\.example\\.com/"
          }
        },
        "name" : "app/node02",
        "path" : {
          "data" : "/var/lib/elasticsearch/data",
          "work" : "/var/lib/elasticsearch",
          "home" : "/usr/share/elasticsearch",
          "conf" : "/etc/elasticsearch",
          "logs" : "/var/log/elasticsearch"
        },
        "cloud" : {
          "aws" : {
            "region" : "us-west-1"
          },
          "node" : {
            "auto_attributes" : "true"
          }
        },
        "config" : "/etc/elasticsearch/elasticsearch.yml",
        "cluster" : {
          "name" : "app"
        },
        "indices" : {
          "cache" : {
            "filter" : {
              "size" : "30%"
            }
          },
          "memory" : {
            "index_buffer_size" : "50%"
          }
        },
        "discovery" : {
          "type" : "ec2",
          "ec2" : {
            "groups" : "appfeed"
          }
        },
        "foreground" : "yes"
      }
    }
  }
}

@imotov imotov added the >bug label Apr 21, 2015
@imotov
Copy link
Contributor

imotov commented Apr 21, 2015

@tomas-edwardsson thanks! We can reproduce it now. An important part that was missing is that this issue only occurs with EC2 discovery plugin.

This was referenced Apr 21, 2015
Merged
Closed
Closed
dadoonet added a commit to elastic/elasticsearch-cloud-gce that referenced this issue Apr 22, 2015
@dadoonet
Fix non working update dynamic settings
ebe75a3 
Described in elastic/elasticsearch#10614, it's not possible with cloud discovery plugin to update dynamic settings anymore.

```sh
curl -XPUT localhost:9200/_cluster/settings -d '{
    "persistent" : {
        "discovery.zen.minimum_master_nodes" : 3
    },
    "transient" : {
        "discovery.zen.minimum_master_nodes" : 3
    }
}'
```

gives

```json
{"acknowledged":true,"persistent":{},"transient":{}}
```

Closes #47.

(cherry picked from commit 9df33a3)
(cherry picked from commit a40016e)
dadoonet added a commit to elastic/elasticsearch-cloud-gce that referenced this issue Apr 22, 2015
@dadoonet
Fix non working update dynamic settings
a40016e 
Described in elastic/elasticsearch#10614, it's not possible with cloud discovery plugin to update dynamic settings anymore.

```sh
curl -XPUT localhost:9200/_cluster/settings -d '{
    "persistent" : {
        "discovery.zen.minimum_master_nodes" : 3
    },
    "transient" : {
        "discovery.zen.minimum_master_nodes" : 3
    }
}'
```

gives

```json
{"acknowledged":true,"persistent":{},"transient":{}}
```

Closes #47.

(cherry picked from commit 9df33a3)
dadoonet added a commit to elastic/elasticsearch-cloud-gce that referenced this issue Apr 22, 2015
@dadoonet
Fix non working update dynamic settings
9df33a3 
Described in elastic/elasticsearch#10614, it's not possible with cloud discovery plugin to update dynamic settings anymore.

```sh
curl -XPUT localhost:9200/_cluster/settings -d '{
    "persistent" : {
        "discovery.zen.minimum_master_nodes" : 3
    },
    "transient" : {
        "discovery.zen.minimum_master_nodes" : 3
    }
}'
```

gives

```json
{"acknowledged":true,"persistent":{},"transient":{}}
```

Closes #47.
imotov added a commit to imotov/elasticsearch that referenced this issue Apr 24, 2015
@imotov
Internal: prevent injection of unannotated dynamic settings
22c88aa 
Dynamic settings has to be injected into constructor with either @ClusterDynamicSettings or @IndexDynamicSettings. If annotations are not specified an empty instance of Dynamic Settings is injected that can lead to difficult to discover errors such as elastic#10614. This commit will make any attempt to inject unannotated dynamic settings to generate a giuce error.
@imotov imotov mentioned this issue Apr 24, 2015
Merged
dadoonet added a commit to elastic/elasticsearch-cloud-azure that referenced this issue Apr 29, 2015
@dadoonet
Fix non working update dynamic settings
82bea69 
Described in elastic/elasticsearch#10614, it's not possible with cloud discovery plugin to update dynamic settings anymore.

```sh
curl -XPUT localhost:9200/_cluster/settings -d '{
    "persistent" : {
        "discovery.zen.minimum_master_nodes" : 3
    },
    "transient" : {
        "discovery.zen.minimum_master_nodes" : 3
    }
}'
```

gives

```json
{"acknowledged":true,"persistent":{},"transient":{}}
```

Closes #80.
(cherry picked from commit 7749055)
dadoonet added a commit to elastic/elasticsearch-cloud-azure that referenced this issue Apr 29, 2015
@dadoonet
Fix non working update dynamic settings
7749055 
Described in elastic/elasticsearch#10614, it's not possible with cloud discovery plugin to update dynamic settings anymore.

```sh
curl -XPUT localhost:9200/_cluster/settings -d '{
    "persistent" : {
        "discovery.zen.minimum_master_nodes" : 3
    },
    "transient" : {
        "discovery.zen.minimum_master_nodes" : 3
    }
}'
```

gives

```json
{"acknowledged":true,"persistent":{},"transient":{}}
```

Closes #80.
dadoonet added a commit to elastic/elasticsearch-cloud-azure that referenced this issue Apr 29, 2015
@dadoonet
Fix non working update dynamic settings
2c1f209 
Described in elastic/elasticsearch#10614, it's not possible with cloud discovery plugin to update dynamic settings anymore.

```sh
curl -XPUT localhost:9200/_cluster/settings -d '{
    "persistent" : {
        "discovery.zen.minimum_master_nodes" : 3
    },
    "transient" : {
        "discovery.zen.minimum_master_nodes" : 3
    }
}'
```

gives

```json
{"acknowledged":true,"persistent":{},"transient":{}}
```

Closes #80.
(cherry picked from commit 7749055)
(cherry picked from commit 82bea69)

Conflicts:
	src/main/java/org/elasticsearch/discovery/azure/AzureDiscovery.java
dadoonet added a commit to elastic/elasticsearch-cloud-aws that referenced this issue Apr 29, 2015
@dadoonet
Fix non working update dynamic settings
e00c099 
Described in elastic/elasticsearch#10614, it's not possible with cloud discovery plugin to update dynamic settings anymore.

```sh
curl -XPUT localhost:9200/_cluster/settings -d '{
    "persistent" : {
        "discovery.zen.minimum_master_nodes" : 3
    },
    "transient" : {
        "discovery.zen.minimum_master_nodes" : 3
    }
}'
```

gives

```json
{"acknowledged":true,"persistent":{},"transient":{}}
```

This patch makes that working again.

(cherry picked from commit 37d6897)
(cherry picked from commit 7a8c7b0)

Conflicts:
	src/main/java/org/elasticsearch/discovery/ec2/Ec2Discovery.java
dadoonet added a commit to elastic/elasticsearch-cloud-aws that referenced this issue Apr 29, 2015
@dadoonet
Fix non working update dynamic settings
7a8c7b0 
Described in elastic/elasticsearch#10614, it's not possible with cloud discovery plugin to update dynamic settings anymore.

```sh
curl -XPUT localhost:9200/_cluster/settings -d '{
    "persistent" : {
        "discovery.zen.minimum_master_nodes" : 3
    },
    "transient" : {
        "discovery.zen.minimum_master_nodes" : 3
    }
}'
```

gives

```json
{"acknowledged":true,"persistent":{},"transient":{}}
```

This patch makes that working again.

(cherry picked from commit 37d6897)
@dadoonet
Copy link
Member

Closing this issue as it has been fixed in cloud-aws, cloud-azure and cloud-gce plugins (versions not released yet)

imotov added a commit to imotov/elasticsearch that referenced this issue May 8, 2015
@imotov
Internal: prevent injection of unannotated dynamic settings
d2844e0 
Dynamic settings has to be injected into constructor with either @ClusterDynamicSettings or @IndexDynamicSettings. If annotations are not specified an empty instance of Dynamic Settings is injected that can lead to difficult to discover errors such as elastic#10614. This commit will make any attempt to inject unannotated dynamic settings to generate a giuce error.
@pmop pmop mentioned this issue Mar 9, 2018
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dadoonet dadoonet

Labels
>bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@dadoonet @tomas-edwardsson @imotov