[DOCS] Reloadable Secure Settings (#31713)

Docs on reloadable secure settings for plugins #29135 .

Author: albertzaharovits

docs/plugins/discovery-ec2.asciidoc

@@ -37,7 +37,6 @@ bin/elasticsearch-keystore add discovery.ec2.secret_key
 The following are the available discovery settings. All should be prefixed with `discovery.ec2.`.
 Those that must be stored in the keystore are marked as `Secure`.
 
-
 `access_key`::
 
     An ec2 access key. The `secret_key` setting must also be specified. (Secure)
@@ -122,6 +121,10 @@ Defaults to `private_ip`.
     How long the list of hosts is cached to prevent further requests to the AWS API.
     Defaults to `10s`.
 
+*All* secure settings of this plugin are {ref}/secure-settings.html#reloadable-secure-settings[reloadable].
+After you reload the settings, an aws sdk client with the latest settings
+from the keystore will be used.
+
 [IMPORTANT]
 .Binding the network host
 ==============================================

docs/plugins/repository-azure.asciidoc

@@ -11,7 +11,7 @@ include::install_remove.asciidoc[]
 ==== Azure Repository
 
 To enable Azure repositories, you have first to define your azure storage settings as
-{ref}/secure-settings.html[secured settings]:
+{ref}/secure-settings.html[secure settings], before starting up the node:
 
 [source,sh]
 ----------------------------------------------------------------
@@ -20,6 +20,7 @@ bin/elasticsearch-keystore add azure.client.default.key
 ----------------------------------------------------------------
 
 Where `account` is the azure account name and `key` the azure secret key.
+These settings are used by the repository's internal azure client.
 
 Note that you can also define more than one account:
 
@@ -31,7 +32,18 @@ bin/elasticsearch-keystore add azure.client.secondary.account
 bin/elasticsearch-keystore add azure.client.secondary.key
 ----------------------------------------------------------------
 
-`default` is the default account name which will be used by a repository unless you set an explicit one.
+`default` is the default account name which will be used by a repository,
+unless you set an explicit one in the
+<<repository-azure-repository-settings, repository settings>>.
+
+Both `account` and `key` storage settings are
+{ref}/secure-settings.html#reloadable-secure-settings[reloadable]. After you
+reload the settings, the internal azure clients, which are used to transfer the
+snapshot, will utilize the latest settings from the keystore.
+
+NOTE: In progress snapshot/restore jobs will not be preempted by a *reload*
+of the storage secure settings. They will complete using the client as it was built
+when the operation started.
 
 You can set the client side timeout to use when making any single request. It can be defined globally, per account or both.
 It's not set by default which means that Elasticsearch is using the

docs/plugins/repository-gcs.asciidoc

@@ -112,6 +112,15 @@ PUT _snapshot/my_gcs_repository
 // CONSOLE
 // TEST[skip:we don't have gcs setup while testing this]
 
+The `credentials_file` settings are {ref}/secure-settings.html#reloadable-secure-settings[reloadable].
+After you reload the settings, the internal `gcs` clients, used to transfer the
+snapshot contents, will utilize the latest settings from the keystore.
+
+
+NOTE: In progress snapshot/restore jobs will not be preempted by a *reload*
+of the client's `credentials_file` settings. They will complete using the client
+as it was built when the operation started.
+
 [[repository-gcs-client]]
 ==== Client Settings
 

docs/plugins/repository-s3.asciidoc

@@ -35,9 +35,9 @@ PUT _snapshot/my_s3_repository
 ==== Client Settings
 
 The client used to connect to S3 has a number of settings available. Client setting names are of
-the form `s3.client.CLIENT_NAME.SETTING_NAME` and specified inside `elasticsearch.yml`.  The
-default client name looked up by a `s3` repository is called `default`, but can be customized
-with the repository setting `client`. For example:
+the form `s3.client.CLIENT_NAME.SETTING_NAME`. The default client name, which is looked up by
+an `s3` repository, is called `default`. It can be modified using the
+<<repository-s3-repository, repository setting>> `client`. For example:
 
 [source,js]
 ----
@@ -53,17 +53,31 @@ PUT _snapshot/my_s3_repository
 // CONSOLE
 // TEST[skip:we don't have s3 setup while testing this]
 
-Some settings are sensitive and must be stored in the {ref}/secure-settings.html[elasticsearch keystore].
-For example, to use explicit AWS access keys:
+Most client settings are specified inside `elasticsearch.yml`, but some are
+sensitive and must be stored in the {ref}/secure-settings.html[elasticsearch keystore].
+
+For example, before you start the node, run these commands to add AWS access
+key settings to the keystore:
 
 [source,sh]
 ----
 bin/elasticsearch-keystore add s3.client.default.access_key
 bin/elasticsearch-keystore add s3.client.default.secret_key
 ----
 
-The following are the available client settings. Those that must be stored in the keystore
-are marked as `Secure`.
+*All* client secure settings of this plugin are
+{ref}/secure-settings.html#reloadable-secure-settings[reloadable]. After you
+reload the settings, the internal `s3` clients, used to transfer the snapshot
+contents, will utilize the latest settings from the keystore. Any existing `s3`
+repositories, as well as any newly created ones, will pick up the new values
+stored in the keystore.
+
+NOTE: In progress snapshot/restore tasks will not be preempted by a *reload*
+of the client's secure settings. The task will complete using the client as it
+was built when the operation started.
+
+The following is the list of all the available client settings.
+Those that must be stored in the keystore are marked as `Secure` and are *reloadable*.
 
 `access_key`::
 

docs/reference/setup/secure-settings.asciidoc

@@ -75,3 +75,34 @@ To remove a setting from the keystore, use the `remove` command:
 bin/elasticsearch-keystore remove the.setting.name.to.remove
 ----------------------------------------------------------------
 
+[float]
+[[reloadable-secure-settings]]
+=== Reloadable secure settings
+
+Just like the settings values in `elasticsearch.yml`, changes to the
+keystore contents are not automatically applied to the running
+elasticsearch node. Re-reading settings requires a node restart.
+However, certain secure settings are marked as *reloadable*. Such settings
+can be re-read and applied on a running node.
+
+The values of all secure settings, *reloadable* or not, must be identical
+across all cluster nodes. After making the desired secure settings changes,
+using the `bin/elasticsearch-keystore add` command, call:
+[source,js]
+----
+POST _nodes/reload_secure_settings
+{
+  "secure_settings_password": ""
+}
+----
+// CONSOLE
+This API will decrypt and re-read the entire keystore, on every cluster node,
+but only the *reloadable* secure settings will be applied. Changes to other
+settings will not go into effect until the next restart. Once the call returns,
+the reload has been completed, meaning that all internal datastructures dependent
+on these settings have been changed. Everything should look as if the settings
+had the new value from the start.
+
+When changing multiple *reloadable* secure settings, modify all of them, on
+each cluster node, and then issue a `reload_secure_settings` call, instead
+of reloading after each modification.