diff --git a/content/en/docs/tasks/administer-cluster/encrypt-data.md b/content/en/docs/tasks/administer-cluster/encrypt-data.md
index c48f9ee2dab2f..acefd13c46819 100644
--- a/content/en/docs/tasks/administer-cluster/encrypt-data.md
+++ b/content/en/docs/tasks/administer-cluster/encrypt-data.md
@@ -7,33 +7,56 @@ min-kubernetes-server-version: 1.13
---
+Some of the APIs in Kubernetes, such as {{< glossary_tooltip text="Secret" term_id="secret" >}},
+support at-rest encryption. This at-rest encryption is additional to any system-level
+encryption for the etcd cluster or hosts where the kube-apiserver stores data persistently.
+
This page shows how to enable and configure encryption of secret data at rest.
+{{< note >}}
+This task covers encryption for data stored using the
+{{< glossary_tooltip text="Kubernetes API" term_id="kubernetes-api" >}}.
+
+If you want to encrypt data in filesystems that are mounted into containers, you instead need
+to use a storage integration that provides encrypted
+{{< glossary_tooltip text="volumes" term_id="volume" >}}.
+{{< /note >}}
+
## {{% heading "prerequisites" %}}
* {{< include "task-tutorial-prereqs.md" >}} {{< version-check >}}
-* etcd v3.0 or later is required
+* Your cluster's control plane must use etcd v3.x (major version 3, any minor version)
## Configuration and determining whether encryption at rest is already enabled
+By default, the API server uses `identity` provider is used to protect secrets in etcd, which
+provides no encryption.
+
The `kube-apiserver` process accepts an argument `--encryption-provider-config`
that controls how API data is encrypted in etcd.
The configuration is provided as an API named
[`EncryptionConfiguration`](/docs/reference/config-api/apiserver-encryption.v1/).
-An example configuration is provided below.
+You can see an example configuration
+in [Encryption at rest configuration](#understanding-the-encryption-at-rest-configuration).
{{< caution >}}
-**IMPORTANT:** For high-availability configurations (with two or more control plane nodes), the
-encryption configuration file must be the same! Otherwise, the `kube-apiserver` component cannot
-decrypt data stored in the etcd.
+For cluster configurations with two or more control plane nodes, the encryption configuration
+must be identical across each control plane node.
+
+If there is a difference in the encryption provider configuration, this may well mean
+that the kube-apiserver can't decrypt data stored inside the key-value store (potentially
+leading to further problems).
{{< /caution >}}
-## Understanding the encryption at rest configuration.
+## Encryption-at-rest configuration {#understanding-the-encryption-at-rest-configuration}
+
+Here is an example EncryptionConfiguration file for the kube-apiserver:
```yaml
+---
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
@@ -59,10 +82,15 @@ resources:
secret: YWJjZGVmZ2hpamtsbW5vcHFyc3R1dnd4eXoxMjM0NTY=
```
-Each `resources` array item is a separate config and contains a complete configuration. The
-`resources.resources` field is an array of Kubernetes resource names (`resource` or `resource.group`)
+Each `resources` array item is a separate configuration and contains a complete configuration. The
+`resources.resources` field is an array of Kubernetes resource types that the API server should
+encrypt.
+You can specify resource types as a simple type based on its URL representation (for example,
+`secrets` for the [Secret API](/docs/reference/kubernetes-api/config-and-storage-resources/secret-v1/)).
+You can also specify encryption for an API group (for example: `events.k8s.io`)
that should be encrypted. The `providers` array is an ordered list of the possible encryption
-providers.
+providers. Only one provider type may be specified per top-level entry
+(for example: you can specify `identity` or `aescbc`, but not both in the same item).
Only one provider type may be specified per entry (`identity` or `aescbc` may be provided,
but not both in the same item).
@@ -71,28 +99,35 @@ resources from storage, each provider that matches the stored data attempts in o
data. If no provider can read the stored data due to a mismatch in format or secret key, an error
is returned which prevents clients from accessing that resource.
-For more detailed information about the `EncryptionConfiguration` struct, please refer to the
-[encryption configuration API](/docs/reference/config-api/apiserver-encryption.v1/).
+For more detailed information about the `EncryptionConfiguration` for the kube-apiserver,
+refer to the
+[encryption configuration reference](/docs/reference/config-api/apiserver-encryption.v1/).
-{{< caution >}}
-If any resource is not readable via the encryption config (because keys were changed),
-the only recourse is to delete that key from the underlying etcd directly. Calls that attempt to
-read that resource will fail until it is deleted or a valid decryption key is provided.
-{{< /caution >}}
+{{< note >}}
+If any resource is not readable via the encryption configuration (because keys were changed),
+and you cannot restore a working configuration, your only recourse is to delete that key from
+the underlying etcd directly. Calls that attempt to read that resource will fail until it is
+deleted or a valid decryption key is provided.
+{{< /note >}}
-### Providers:
+## Providers
{{< table caption="Providers for Kubernetes encryption at rest" >}}
-Name | Encryption | Strength | Speed | Key Length | Other Considerations
+Name | Encryption | Strength | Speed | Key length | Other considerations
-----|------------|----------|-------|------------|---------------------
-`identity` | None | N/A | N/A | N/A | Resources written as-is without encryption. When set as the first provider, the resource will be decrypted as new values are written.
-`secretbox` | XSalsa20 and Poly1305 | Strong | Faster | 32-byte | A newer standard and may not be considered acceptable in environments that require high levels of review.
+`identity` | None | N/A | N/A | N/A | Resources written as-is without encryption. When set as the first provider, the resource will be decrypted as new values are written. Existing encrypted resource are **not** automatically overwritten with the plaintext data.
`aesgcm` | AES-GCM with random nonce | Must be rotated every 200k writes | Fastest | 16, 24, or 32-byte | Is not recommended for use except when an automated key rotation scheme is implemented.
`aescbc` | AES-CBC with PKCS#7 padding | Weak | Fast | 32-byte | Not recommended due to CBC's vulnerability to padding oracle attacks.
-`kms` | Uses envelope encryption scheme: Data is encrypted by data encryption keys (DEKs) using AES-CBC with PKCS#7 padding, DEKs are encrypted by key encryption keys (KEKs) according to configuration in Key Management Service (KMS) | Strongest | Fast | 32-bytes | The recommended choice for using a third party tool for key management. Simplifies key rotation, with a new DEK generated for each encryption, and KEK rotation controlled by the user. [Configure the KMS provider](/docs/tasks/administer-cluster/kms-provider/)
+`kms` | Uses envelope encryption scheme: Data is encrypted by data encryption keys (DEKs) using AES-CBC with PKCS#7 padding; DEKs are encrypted by key encryption keys (KEKs) according to configuration in Key Management Service (KMS) | Strongest | Fast | 32-bytes | The recommended choice for using a third party tool for key management. Simplifies key rotation, with a new DEK generated for each encryption, and KEK rotation controlled by the user. See [Configure the KMS provider](/docs/tasks/administer-cluster/kms-provider/). May incur extra costs for the external key management.
+`secretbox` | XSalsa20 and Poly1305 | Strong | Faster | 32-byte | Uses relatively new encryption technologies that may not be considered acceptable in environments that require high levels of review.
+
+You can configure multiple provides, and each provider (other than `identity`, which does not encrypt) supports multiple keys.
+For encryption, the API server uses the first configured key from the first provider.
+For decryption, the API server tries each key in order for decryption, stopping when decryption succeeds.
+The API server can detect which provider to use for decryption based on metadata that accompanies the stored ciphertext.
-Each provider supports multiple keys - the keys are tried in order for decryption, and if the provider
-is the first provider, the first key is used for encryption.
+Storing the raw encryption key in the EncryptionConfig only moderately improves your security posture, compared to no encryption.
+Please use `kms` provider for additional security.
{{< caution >}}
@@ -115,9 +150,10 @@ retrieve the plaintext values, providing a higher level of security than locally
## Encrypting your data
-Create a new encryption config file:
+Create a new encryption configuration file. The contents should be similar to:
```yaml
+---
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
@@ -127,11 +163,12 @@ resources:
- aescbc:
keys:
- name: key1
+ # See the following text for more details about this value.
secret:
- - identity: {}
+ - identity: {} # this fallback allows reading unencrypted secrets
```
-To create a new Secret, perform the following steps:
+To create a new encryption secret, perform the following steps:
1. Generate a 32-byte random key and base64 encode it. If you're on Linux or macOS, run the following command:
@@ -139,21 +176,33 @@ To create a new Secret, perform the following steps:
head -c 32 /dev/urandom | base64
```
-1. Place that value in the `secret` field of the `EncryptionConfiguration` struct.
+1. Place that value in the `secret` field of the encryption configuration file.
1. Set the `--encryption-provider-config` flag on the `kube-apiserver` to point to
the location of the config file.
-1. Restart your API server.
+1. Restart the API server.
+
+If you have multiple API servers in your cluster, you should deploy the
+changes in turn to each API server.
+
+
+{{< note >}}
+Keep the encryption secret confidential, including whilst you generate it and
+ideally even after you are no longer actively using it.
+{{< /note >}}
{{< caution >}}
-Your config file contains keys that can decrypt the contents in etcd, so you must properly restrict
-permissions on your control-plane nodes so only the user who runs the `kube-apiserver` can read it.
+This configuration file contains keys that can decrypt content in etcd.
+If the configuration file contains any key material, you must properly
+restrict permissions on all your control plane hosts so only the user
+who runs the kube-apiserver can read this configuration.
{{< /caution >}}
+
## Verifying that data is encrypted
-Data is encrypted when written to etcd. After restarting your `kube-apiserver`, any newly created or
-updated Secret should be encrypted when stored. To check this, you can use the `etcdctl` command line
-program to retrieve the contents of your Secret.
+Data is encrypted when written to etcd. After restarting your `kube-apiserver`,
+any newly created or updated Secret should be encrypted when stored. To check this,
+you can use the `etcdctl` command line program to retrieve the contents of your Secret.
1. Create a new Secret called `secret1` in the `default` namespace:
@@ -161,7 +210,7 @@ program to retrieve the contents of your Secret.
kubectl create secret generic secret1 -n default --from-literal=mykey=mydata
```
-1. Using the `etcdctl` command line, read that Secret out of etcd:
+1. Using the `etcdctl` command line tool, read that Secret out of etcd:
`ETCDCTL_API=3 etcdctl get /registry/secrets/default/secret1 [...] | hexdump -C`
@@ -182,9 +231,13 @@ program to retrieve the contents of your Secret.
## Ensure all Secrets are encrypted
-Since Secrets are encrypted on write, performing an update on a Secret will encrypt that content.
+You have configured your cluster so that Secrets are encrypted on write. Performing
+an update on Secret will encrypt that content at rest.
+
+You can make this change across all Secrets in your cluster:
```shell
+# Run this as an administrator that can read and write all Secrets
kubectl get secrets --all-namespaces -o json | kubectl replace -f -
```
@@ -192,31 +245,66 @@ The command above reads all Secrets and then updates them to apply server side e
{{< note >}}
If an error occurs due to a conflicting write, retry the command.
-For larger clusters, you may wish to subdivide the secrets by namespace or script an update.
+For larger clusters, you may wish to subdivide the Secrets by namespace, or script an update.
{{< /note >}}
+### Cleanup {cleanup-all-secrets-encrypted}
+
+Once all Secrets in your cluster are encrypted, you can remove the `identity`
+part of the encryption configuration. For example:
+
+```yaml
+---
+apiVersion: apiserver.config.k8s.io/v1
+kind: EncryptionConfiguration
+resources:
+ - resources:
+ - secrets
+ providers:
+ - aescbc:
+ keys:
+ - name: key1
+ secret:
+ - identity: {} # REMOVE THIS LINE
+```
+
+and then restart each API server in turn. This change prevents the API server
+from accessing a plain-text Secret, so you **must** first ensure that there are
+no resources of that type (`secrets`, for this example) before performing the
+clean up steps.
+
## Rotating a decryption key
Changing a Secret without incurring downtime requires a multi-step operation, especially in
the presence of a highly-available deployment where multiple `kube-apiserver` processes are running.
-1. Generate a new key and add it as the second key entry for the current provider on all servers
-1. Restart all `kube-apiserver` processes to ensure each server can decrypt using the new key
-1. Make the new key the first entry in the `keys` array so that it is used for encryption in the config
-1. Restart all `kube-apiserver` processes to ensure each server now encrypts using the new key
-1. Run `kubectl get secrets --all-namespaces -o json | kubectl replace -f -` to encrypt all
- existing Secrets with the new key
-1. Remove the old decryption key from the config after you have backed up etcd with the new key in use
- and updated all Secrets
+
+1. Generate a new key and add it as the second key entry for the current provider on all
+ control plane hosts (strictly speaking: on all hosts that run a kube-apiserver).
+2. Restart all `kube-apiserver` processes, to ensure each control plane host can decrypt
+ any data that are encrypted with the new key
+ (this is required for a replicated control plane; you can skip this step if you only run a
+ single `kube-apiserver`).
+3. Make a secure backup of the new encryption key. If you lose all copies of this key you would
+ need to delete all the resources were encrypted under the lost key, and workloads may not
+ operate as expected during the time that at-rest encryption is broken.
+4. Make the new key the first entry in the `keys` array so that it is used for encryption-at-rest
+ for new writes
+5. Restart all `kube-apiserver` processes to ensure each control plane host now encrypts using the new key
+6. As a privileged user, run `kubectl get secrets --all-namespaces -o json | kubectl replace -f -`
+ to encrypt all existing Secrets with the new key
+7. After you have updated all existing Secrets to use the new key and have made a secure backup of the
+ new key, remove the old decryption key from the configuration.
When running a single `kube-apiserver` instance, step 2 may be skipped.
## Decrypting all data
-To disable encryption at rest, place the `identity` provider as the first entry in the config
-and restart all `kube-apiserver` processes.
+To disable encryption at rest, place the `identity` provider as the first
+entry your encryption configuration file:
```yaml
+---
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
@@ -227,7 +315,7 @@ resources:
- aescbc:
keys:
- name: key1
- secret:
+ secret: # keep this in place
```
Then run the following command to force decrypt
@@ -236,6 +324,11 @@ all Secrets:
```shell
kubectl get secrets --all-namespaces -o json | kubectl replace -f -
```
+to force all Secrets to be decrypted.
+
+Once you have replaced all existing encrypted resources with backing data that
+don't use encryption, you can remove the encryption settings from the
+kube-apiserver.
## {{% heading "whatsnext" %}}