Setting persistent settings causes transient settings not to be applied

[dakrone]

Given the following scenario in a multi-node cluster:

• apply some set of transient settings
• ensure settings are applied
• apply addition persistent setting
• transient settings are ignored but still available in cluster state

This is reproducible with the following:

• Create 6 nodes
• Create a "test" index (default settings)
• Apply transient settings to exclude 3 of the nodes by _name with allocation filtering
• Ensure shards have moved off of the 3 excluded nodes
• Apply a persistent setting (I used cluster.info.update.interval but any setting will work)
• Shards no longer conform to the transient allocation filtering settings
• Settings are still shown in the cluster state

This reproduces on master, 6.x, and 6.1, but does not on 6.0 and earlier.

Attached is the (very quickly hacked together) test that reproduces this:
ClusterBugIT.java.txt

[dakrone]

Looking into the root cause for this, it appears that the affixMapUpdateConsumer that's added in FilterAllocationDecider is called with an empty map when the persistent setting is added, so it's not passing the existing transient settings to the consumer