diff --git a/content/en/docs/tasks/administer-cluster/encrypt-data.md b/content/en/docs/tasks/administer-cluster/encrypt-data.md index c48f9ee2dab2f..acefd13c46819 100644 --- a/content/en/docs/tasks/administer-cluster/encrypt-data.md +++ b/content/en/docs/tasks/administer-cluster/encrypt-data.md @@ -7,33 +7,56 @@ min-kubernetes-server-version: 1.13 --- +Some of the APIs in Kubernetes, such as {{< glossary_tooltip text="Secret" term_id="secret" >}}, +support at-rest encryption. This at-rest encryption is additional to any system-level +encryption for the etcd cluster or hosts where the kube-apiserver stores data persistently. + This page shows how to enable and configure encryption of secret data at rest. +{{< note >}} +This task covers encryption for data stored using the +{{< glossary_tooltip text="Kubernetes API" term_id="kubernetes-api" >}}. + +If you want to encrypt data in filesystems that are mounted into containers, you instead need +to use a storage integration that provides encrypted +{{< glossary_tooltip text="volumes" term_id="volume" >}}. +{{< /note >}} + ## {{% heading "prerequisites" %}} * {{< include "task-tutorial-prereqs.md" >}} {{< version-check >}} -* etcd v3.0 or later is required +* Your cluster's control plane must use etcd v3.x (major version 3, any minor version) ## Configuration and determining whether encryption at rest is already enabled +By default, the API server uses `identity` provider is used to protect secrets in etcd, which +provides no encryption. + The `kube-apiserver` process accepts an argument `--encryption-provider-config` that controls how API data is encrypted in etcd. The configuration is provided as an API named [`EncryptionConfiguration`](/docs/reference/config-api/apiserver-encryption.v1/). -An example configuration is provided below. +You can see an example configuration +in [Encryption at rest configuration](#understanding-the-encryption-at-rest-configuration). {{< caution >}} -**IMPORTANT:** For high-availability configurations (with two or more control plane nodes), the -encryption configuration file must be the same! Otherwise, the `kube-apiserver` component cannot -decrypt data stored in the etcd. +For cluster configurations with two or more control plane nodes, the encryption configuration +must be identical across each control plane node. + +If there is a difference in the encryption provider configuration, this may well mean +that the kube-apiserver can't decrypt data stored inside the key-value store (potentially +leading to further problems). {{< /caution >}} -## Understanding the encryption at rest configuration. +## Encryption-at-rest configuration {#understanding-the-encryption-at-rest-configuration} + +Here is an example EncryptionConfiguration file for the kube-apiserver: ```yaml +--- apiVersion: apiserver.config.k8s.io/v1 kind: EncryptionConfiguration resources: @@ -59,10 +82,15 @@ resources: secret: YWJjZGVmZ2hpamtsbW5vcHFyc3R1dnd4eXoxMjM0NTY= ``` -Each `resources` array item is a separate config and contains a complete configuration. The -`resources.resources` field is an array of Kubernetes resource names (`resource` or `resource.group`) +Each `resources` array item is a separate configuration and contains a complete configuration. The +`resources.resources` field is an array of Kubernetes resource types that the API server should +encrypt. +You can specify resource types as a simple type based on its URL representation (for example, +`secrets` for the [Secret API](/docs/reference/kubernetes-api/config-and-storage-resources/secret-v1/)). +You can also specify encryption for an API group (for example: `events.k8s.io`) that should be encrypted. The `providers` array is an ordered list of the possible encryption -providers. +providers. Only one provider type may be specified per top-level entry +(for example: you can specify `identity` or `aescbc`, but not both in the same item). Only one provider type may be specified per entry (`identity` or `aescbc` may be provided, but not both in the same item). @@ -71,28 +99,35 @@ resources from storage, each provider that matches the stored data attempts in o data. If no provider can read the stored data due to a mismatch in format or secret key, an error is returned which prevents clients from accessing that resource. -For more detailed information about the `EncryptionConfiguration` struct, please refer to the -[encryption configuration API](/docs/reference/config-api/apiserver-encryption.v1/). +For more detailed information about the `EncryptionConfiguration` for the kube-apiserver, +refer to the +[encryption configuration reference](/docs/reference/config-api/apiserver-encryption.v1/). -{{< caution >}} -If any resource is not readable via the encryption config (because keys were changed), -the only recourse is to delete that key from the underlying etcd directly. Calls that attempt to -read that resource will fail until it is deleted or a valid decryption key is provided. -{{< /caution >}} +{{< note >}} +If any resource is not readable via the encryption configuration (because keys were changed), +and you cannot restore a working configuration, your only recourse is to delete that key from +the underlying etcd directly. Calls that attempt to read that resource will fail until it is +deleted or a valid decryption key is provided. +{{< /note >}} -### Providers: +## Providers {{< table caption="Providers for Kubernetes encryption at rest" >}} -Name | Encryption | Strength | Speed | Key Length | Other Considerations +Name | Encryption | Strength | Speed | Key length | Other considerations -----|------------|----------|-------|------------|--------------------- -`identity` | None | N/A | N/A | N/A | Resources written as-is without encryption. When set as the first provider, the resource will be decrypted as new values are written. -`secretbox` | XSalsa20 and Poly1305 | Strong | Faster | 32-byte | A newer standard and may not be considered acceptable in environments that require high levels of review. +`identity` | None | N/A | N/A | N/A | Resources written as-is without encryption. When set as the first provider, the resource will be decrypted as new values are written. Existing encrypted resource are **not** automatically overwritten with the plaintext data. `aesgcm` | AES-GCM with random nonce | Must be rotated every 200k writes | Fastest | 16, 24, or 32-byte | Is not recommended for use except when an automated key rotation scheme is implemented. `aescbc` | AES-CBC with PKCS#7 padding | Weak | Fast | 32-byte | Not recommended due to CBC's vulnerability to padding oracle attacks. -`kms` | Uses envelope encryption scheme: Data is encrypted by data encryption keys (DEKs) using AES-CBC with PKCS#7 padding, DEKs are encrypted by key encryption keys (KEKs) according to configuration in Key Management Service (KMS) | Strongest | Fast | 32-bytes | The recommended choice for using a third party tool for key management. Simplifies key rotation, with a new DEK generated for each encryption, and KEK rotation controlled by the user. [Configure the KMS provider](/docs/tasks/administer-cluster/kms-provider/) +`kms` | Uses envelope encryption scheme: Data is encrypted by data encryption keys (DEKs) using AES-CBC with PKCS#7 padding; DEKs are encrypted by key encryption keys (KEKs) according to configuration in Key Management Service (KMS) | Strongest | Fast | 32-bytes | The recommended choice for using a third party tool for key management. Simplifies key rotation, with a new DEK generated for each encryption, and KEK rotation controlled by the user. See [Configure the KMS provider](/docs/tasks/administer-cluster/kms-provider/). May incur extra costs for the external key management. +`secretbox` | XSalsa20 and Poly1305 | Strong | Faster | 32-byte | Uses relatively new encryption technologies that may not be considered acceptable in environments that require high levels of review. + +You can configure multiple provides, and each provider (other than `identity`, which does not encrypt) supports multiple keys. +For encryption, the API server uses the first configured key from the first provider. +For decryption, the API server tries each key in order for decryption, stopping when decryption succeeds. +The API server can detect which provider to use for decryption based on metadata that accompanies the stored ciphertext. -Each provider supports multiple keys - the keys are tried in order for decryption, and if the provider -is the first provider, the first key is used for encryption. +Storing the raw encryption key in the EncryptionConfig only moderately improves your security posture, compared to no encryption. +Please use `kms` provider for additional security. {{< caution >}} @@ -115,9 +150,10 @@ retrieve the plaintext values, providing a higher level of security than locally ## Encrypting your data -Create a new encryption config file: +Create a new encryption configuration file. The contents should be similar to: ```yaml +--- apiVersion: apiserver.config.k8s.io/v1 kind: EncryptionConfiguration resources: @@ -127,11 +163,12 @@ resources: - aescbc: keys: - name: key1 + # See the following text for more details about this value. secret: - - identity: {} + - identity: {} # this fallback allows reading unencrypted secrets ``` -To create a new Secret, perform the following steps: +To create a new encryption secret, perform the following steps: 1. Generate a 32-byte random key and base64 encode it. If you're on Linux or macOS, run the following command: @@ -139,21 +176,33 @@ To create a new Secret, perform the following steps: head -c 32 /dev/urandom | base64 ``` -1. Place that value in the `secret` field of the `EncryptionConfiguration` struct. +1. Place that value in the `secret` field of the encryption configuration file. 1. Set the `--encryption-provider-config` flag on the `kube-apiserver` to point to the location of the config file. -1. Restart your API server. +1. Restart the API server. + +If you have multiple API servers in your cluster, you should deploy the +changes in turn to each API server. + + +{{< note >}} +Keep the encryption secret confidential, including whilst you generate it and +ideally even after you are no longer actively using it. +{{< /note >}} {{< caution >}} -Your config file contains keys that can decrypt the contents in etcd, so you must properly restrict -permissions on your control-plane nodes so only the user who runs the `kube-apiserver` can read it. +This configuration file contains keys that can decrypt content in etcd. +If the configuration file contains any key material, you must properly +restrict permissions on all your control plane hosts so only the user +who runs the kube-apiserver can read this configuration. {{< /caution >}} + ## Verifying that data is encrypted -Data is encrypted when written to etcd. After restarting your `kube-apiserver`, any newly created or -updated Secret should be encrypted when stored. To check this, you can use the `etcdctl` command line -program to retrieve the contents of your Secret. +Data is encrypted when written to etcd. After restarting your `kube-apiserver`, +any newly created or updated Secret should be encrypted when stored. To check this, +you can use the `etcdctl` command line program to retrieve the contents of your Secret. 1. Create a new Secret called `secret1` in the `default` namespace: @@ -161,7 +210,7 @@ program to retrieve the contents of your Secret. kubectl create secret generic secret1 -n default --from-literal=mykey=mydata ``` -1. Using the `etcdctl` command line, read that Secret out of etcd: +1. Using the `etcdctl` command line tool, read that Secret out of etcd: `ETCDCTL_API=3 etcdctl get /registry/secrets/default/secret1 [...] | hexdump -C` @@ -182,9 +231,13 @@ program to retrieve the contents of your Secret. ## Ensure all Secrets are encrypted -Since Secrets are encrypted on write, performing an update on a Secret will encrypt that content. +You have configured your cluster so that Secrets are encrypted on write. Performing +an update on Secret will encrypt that content at rest. + +You can make this change across all Secrets in your cluster: ```shell +# Run this as an administrator that can read and write all Secrets kubectl get secrets --all-namespaces -o json | kubectl replace -f - ``` @@ -192,31 +245,66 @@ The command above reads all Secrets and then updates them to apply server side e {{< note >}} If an error occurs due to a conflicting write, retry the command. -For larger clusters, you may wish to subdivide the secrets by namespace or script an update. +For larger clusters, you may wish to subdivide the Secrets by namespace, or script an update. {{< /note >}} +### Cleanup {cleanup-all-secrets-encrypted} + +Once all Secrets in your cluster are encrypted, you can remove the `identity` +part of the encryption configuration. For example: + +```yaml +--- +apiVersion: apiserver.config.k8s.io/v1 +kind: EncryptionConfiguration +resources: + - resources: + - secrets + providers: + - aescbc: + keys: + - name: key1 + secret: + - identity: {} # REMOVE THIS LINE +``` + +and then restart each API server in turn. This change prevents the API server +from accessing a plain-text Secret, so you **must** first ensure that there are +no resources of that type (`secrets`, for this example) before performing the +clean up steps. + ## Rotating a decryption key Changing a Secret without incurring downtime requires a multi-step operation, especially in the presence of a highly-available deployment where multiple `kube-apiserver` processes are running. -1. Generate a new key and add it as the second key entry for the current provider on all servers -1. Restart all `kube-apiserver` processes to ensure each server can decrypt using the new key -1. Make the new key the first entry in the `keys` array so that it is used for encryption in the config -1. Restart all `kube-apiserver` processes to ensure each server now encrypts using the new key -1. Run `kubectl get secrets --all-namespaces -o json | kubectl replace -f -` to encrypt all - existing Secrets with the new key -1. Remove the old decryption key from the config after you have backed up etcd with the new key in use - and updated all Secrets + +1. Generate a new key and add it as the second key entry for the current provider on all + control plane hosts (strictly speaking: on all hosts that run a kube-apiserver). +2. Restart all `kube-apiserver` processes, to ensure each control plane host can decrypt + any data that are encrypted with the new key + (this is required for a replicated control plane; you can skip this step if you only run a + single `kube-apiserver`). +3. Make a secure backup of the new encryption key. If you lose all copies of this key you would + need to delete all the resources were encrypted under the lost key, and workloads may not + operate as expected during the time that at-rest encryption is broken. +4. Make the new key the first entry in the `keys` array so that it is used for encryption-at-rest + for new writes +5. Restart all `kube-apiserver` processes to ensure each control plane host now encrypts using the new key +6. As a privileged user, run `kubectl get secrets --all-namespaces -o json | kubectl replace -f -` + to encrypt all existing Secrets with the new key +7. After you have updated all existing Secrets to use the new key and have made a secure backup of the + new key, remove the old decryption key from the configuration. When running a single `kube-apiserver` instance, step 2 may be skipped. ## Decrypting all data -To disable encryption at rest, place the `identity` provider as the first entry in the config -and restart all `kube-apiserver` processes. +To disable encryption at rest, place the `identity` provider as the first +entry your encryption configuration file: ```yaml +--- apiVersion: apiserver.config.k8s.io/v1 kind: EncryptionConfiguration resources: @@ -227,7 +315,7 @@ resources: - aescbc: keys: - name: key1 - secret: + secret: # keep this in place ``` Then run the following command to force decrypt @@ -236,6 +324,11 @@ all Secrets: ```shell kubectl get secrets --all-namespaces -o json | kubectl replace -f - ``` +to force all Secrets to be decrypted. + +Once you have replaced all existing encrypted resources with backing data that +don't use encryption, you can remove the encryption settings from the +kube-apiserver. ## {{% heading "whatsnext" %}}