diff --git a/content/en/docs/tasks/administer-cluster/encrypt-data.md b/content/en/docs/tasks/administer-cluster/encrypt-data.md index d510caff81a98..b51d9a65b9478 100644 --- a/content/en/docs/tasks/administer-cluster/encrypt-data.md +++ b/content/en/docs/tasks/administer-cluster/encrypt-data.md @@ -1,46 +1,91 @@ --- -title: Encrypting Secret Data at Rest +title: Encrypting Confidential Data at Rest reviewers: - smarterclayton content_type: task -min-kubernetes-server-version: 1.13 --- -This page shows how to enable and configure encryption of secret data at rest. + +Some of the APIs in Kubernetes, such as {{< glossary_tooltip text="Secret" term_id="secret" >}}, +support at-rest encryption. This at-rest encryption is additional to any system-level +encryption for the etcd cluster or hosts where the kube-apiserver stores data persistently. + +This page shows how to enable and configure encryption of API data at rest. + +{{< note >}} +This task covers encryption for data stored using the +{{< glossary_tooltip text="Kubernetes API" term_id="kubernetes-api" >}}. + +If you want to encrypt data in filesystems that are mounted into containers, you instead need +to either: + +- use a storage integration that provides encrypted +{{< glossary_tooltip text="volumes" term_id="volume" >}} +- encrypt the data within your own application +{{< /note >}} ## {{% heading "prerequisites" %}} -* {{< include "task-tutorial-prereqs.md" >}} {{< version-check >}} +* {{< include "task-tutorial-prereqs.md" >}} -* etcd v3.0 or later is required +* Your cluster's control plane **must** use etcd v3.x (major version 3, any minor version) -## Configuration and determining whether encryption at rest is already enabled +## Determining whether encryption at rest is already enabled + +By default, the API server uses the `identity` provider to protect Secrets in etcd. **The default `identity` provider does not provide any confidentiality protection.** The `kube-apiserver` process accepts an argument `--encryption-provider-config` -that controls how API data is encrypted in etcd. -The configuration is provided as an API named +that specifies a path to a configuration file. The contents of that file, if you specify one, control how Kubernetes API data is encrypted in etcd. + +The format of that configuration file is YAML, representing a configuration API kind named [`EncryptionConfiguration`](/docs/reference/config-api/apiserver-encryption.v1/). -An example configuration is provided below. +You can see an example configuration +in [Encryption at rest configuration](#understanding-the-encryption-at-rest-configuration). + +Check whether the `kube-apiserver` process is running with the `--encryption-provider-config` +command line argument is set. If it is not, you do not have encryption at rest is enabled. + +If `--encryption-provider-config` is set, check which resources (such as `secrets`) are +configured for encryption, and what provider is used. +Make sure that the preferred provider for that resource type is **not** `identity`; you +only set `identity` (_no encryption_) as default when you want to disable encryption at +rest. +Then the first-listed provider for a resource is something **other** than `identity`, +then any new information written to resources of that type will be encrypted as configured. + +If you are not sure about the progress of any previous migration to encrypt data at rest, +you must overwrite each resource with a new updated copy, in order to trigger encryption. +See [Encrypting your data](#encrypting-your-data) for details. + +## Encryption-at-rest configuration {#understanding-the-encryption-at-rest-configuration} {{< caution >}} -**IMPORTANT:** For high-availability configurations (with two or more control plane nodes), the -encryption configuration file must be the same! Otherwise, the `kube-apiserver` component cannot -decrypt data stored in the etcd. +For cluster configurations with two or more control plane nodes, the encryption configuration +**must** be identical across each control plane node. + +If there is a difference in the encryption provider configuration, this may well mean +that the kube-apiserver can't decrypt data stored inside the key-value store (potentially +leading to further problems). {{< /caution >}} -## Understanding the encryption at rest configuration. +Here is an example EncryptionConfiguration file for the kube-apiserver: ```yaml +--- +# +# CAUTION: this is an example configuration. +# Do not use this for your own cluster! +# apiVersion: apiserver.config.k8s.io/v1 kind: EncryptionConfiguration resources: - resources: - secrets providers: - - identity: {} + - identity: {} # default to NO encryption - aesgcm: keys: - name: key1 @@ -59,10 +104,15 @@ resources: secret: YWJjZGVmZ2hpamtsbW5vcHFyc3R1dnd4eXoxMjM0NTY= ``` -Each `resources` array item is a separate config and contains a complete configuration. The -`resources.resources` field is an array of Kubernetes resource names (`resource` or `resource.group`) +Each `resources` array item is a separate configuration and contains a complete configuration. The +`resources.resources` field is an array of Kubernetes resource types that the API server should +encrypt. +You can specify resource types as a simple type based on its URL representation (for example, +`secrets` for the [Secret API](/docs/reference/kubernetes-api/config-and-storage-resources/secret-v1/)). +You can also specify encryption for an API group (for example: `events.k8s.io`) that should be encrypted. The `providers` array is an ordered list of the possible encryption -providers. +providers. Only one provider type may be specified per top-level entry +(for example: you can specify `identity` or `aescbc`, but not both in the same item). Only one provider type may be specified per entry (`identity` or `aescbc` may be provided, but not both in the same item). @@ -71,39 +121,39 @@ resources from storage, each provider that matches the stored data attempts in o data. If no provider can read the stored data due to a mismatch in format or secret key, an error is returned which prevents clients from accessing that resource. -For more detailed information about the `EncryptionConfiguration` struct, please refer to the -[encryption configuration API](/docs/reference/config-api/apiserver-encryption.v1/). +For more detailed information about the `EncryptionConfiguration` for the kube-apiserver, +refer to the +[encryption configuration reference](/docs/reference/config-api/apiserver-encryption.v1/). -{{< caution >}} -If any resource is not readable via the encryption config (because keys were changed), -the only recourse is to delete that key from the underlying etcd directly. Calls that attempt to -read that resource will fail until it is deleted or a valid decryption key is provided. -{{< /caution >}} +{{< note >}} +If any resource is not readable via the encryption configuration (because keys were changed), +and you cannot restore a working configuration, your only recourse is to delete that entry from +the underlying etcd directly. -### Providers: +Any calls to the Kubernetes API that attempt to read that resource will fail until it is deleted +or a valid decryption key is provided. +{{< /note >}} + +## Providers {{< table caption="Providers for Kubernetes encryption at rest" >}} -Name | Encryption | Strength | Speed | Key Length | Other Considerations +Name | Encryption | Strength | Speed | Key length | Other considerations -----|------------|----------|-------|------------|--------------------- -`identity` | None | N/A | N/A | N/A | Resources written as-is without encryption. When set as the first provider, the resource will be decrypted as new values are written. -`secretbox` | XSalsa20 and Poly1305 | Strong | Faster | 32-byte | A newer standard and may not be considered acceptable in environments that require high levels of review. -`aesgcm` | AES-GCM with random nonce | Must be rotated every 200k writes | Fastest | 16, 24, or 32-byte | Is not recommended for use except when an automated key rotation scheme is implemented. -`aescbc` | AES-CBC with [PKCS#7](https://datatracker.ietf.org/doc/html/rfc2315) padding | Weak | Fast | 32-byte | Not recommended due to CBC's vulnerability to padding oracle attacks. -`kms` | Uses envelope encryption scheme: Data is encrypted by data encryption keys (DEKs) using AES-CBC with [PKCS#7](https://datatracker.ietf.org/doc/html/rfc2315) padding, DEKs are encrypted by key encryption keys (KEKs) according to configuration in Key Management Service (KMS) | Strongest | Fast | 32-bytes | The recommended choice for using a third party tool for key management. Simplifies key rotation, with a new DEK generated for each encryption, and KEK rotation controlled by the user. [Configure the KMS provider](/docs/tasks/administer-cluster/kms-provider/) - -Each provider supports multiple keys - the keys are tried in order for decryption, and if the provider -is the first provider, the first key is used for encryption. +`identity` | **None** | N/A | N/A | N/A | Resources written as-is without encryption. When set as the first provider, the resource will be decrypted as new values are written. Existing encrypted resource are **not** automatically overwritten with the plaintext data. +`aescbc` | AES-CBC with [PKCS#7](https://datatracker.ietf.org/doc/html/rfc2315) padding | Weak | Fast | 32-byte | Not recommended due to CBC's vulnerability to padding oracle attacks. Key material held within Kubernetes cluster. +`aesgcm` | AES-GCM with random nonce | Must be rotated every 200,000 writes | Fastest | 16, 24, or 32-byte | Not recommended for use except when an automated key rotation scheme is implemented. Key material held within Kubernetes cluster. +`kms` | Uses envelope encryption scheme: Data is encrypted by data encryption keys (DEKs) using AES-CBC with [PKCS#7](https://datatracker.ietf.org/doc/html/rfc2315) padding; DEKs are encrypted by key encryption keys (KEKs) according to configuration in Key Management Service (KMS) | Strongest | Fast | 32-bytes | The recommended choice for using a third party tool for key management. Simplifies key rotation, with a new DEK generated for each encryption, and KEK rotation controlled by the user. See [Configure the KMS provider](/docs/tasks/administer-cluster/kms-provider/). May incur extra costs for the external key management. +`secretbox` | XSalsa20 and Poly1305 | Strong | Faster | 32-byte | Uses relatively new encryption technologies that may not be considered acceptable in environments that require high levels of review. Key material held within Kubernetes cluster. +{{< /table >}} +You can configure multiple providers, and each provider (other than `identity`, which does not encrypt) supports multiple keys. +For encryption, the API server uses the first configured key from the first provider. +For decryption, the API server tries each key in order for decryption, stopping when decryption succeeds. +The API server can detect which provider to use for decryption based on metadata that accompanies the stored ciphertext. -{{< caution >}} -Storing the raw encryption key in the EncryptionConfig only moderately improves your security -posture, compared to no encryption. Please use `kms` provider for additional security. -{{< /caution >}} By default, the `identity` provider is used to protect Secrets in etcd, which provides no -encryption. `EncryptionConfiguration` was introduced to encrypt Secrets locally, with a locally -managed key. - +encryption. Encrypting Secrets with a locally managed key protects against an etcd compromise, but it fails to protect against a host compromise. Since the encryption keys are stored on the host in the EncryptionConfiguration YAML file, a skilled attacker can access that file and extract the encryption @@ -113,11 +163,54 @@ Envelope encryption creates dependence on a separate key, not stored in Kubernet an attacker would need to compromise etcd, the `kubeapi-server`, and the third-party KMS provider to retrieve the plaintext values, providing a higher level of security than locally stored encryption keys. +{{< caution >}} +Storing the raw encryption key in the EncryptionConfig only moderately improves your security posture, compared to no encryption. +For additional secrecy, consider using the `kms` provider which relies on keys held outside your Kubernetes cluster. +{{< /caution >}} + ## Encrypting your data -Create a new encryption config file: + +To generate a new encryption key: +{{< tabs name="generate_encryption_key" >}} +{{% tab name="Linux" %}} +Generate a 32-byte random key and base64 encode it. You can use this command: +```shell +head -c 32 /dev/urandom | base64 +``` + +You can use `/dev/hwrng` instead of /dev/urandom if you want to +use your PC's built-in hardware entropy source. Not all Linux +devices provide a hardware random generator. +{{% /tab %}} +{{% tab name="macOS" %}} + +Generate a 32-byte random key and base64 encode it. You can use this command: +```shell +head -c 32 /dev/urandom | base64 +``` +{{% /tab %}} +{{% tab name="Windows" %}} +Generate a 32-byte random key and base64 encode it. You can use this command: +```powershell +# Do not run this in a session where you have set a random number +# generator seed. +[Convert]::ToBase64String((1..32|%{[byte](Get-Random -Max 256)})) +``` +{{% /tab %}} +{{< /tabs >}} + + +{{< note >}} +Keep the encryption secret confidential, including whilst you generate it and +ideally even after you are no longer actively using it. +{{< /note >}} + +Create a new encryption configuration file. The contents should be similar to: ```yaml +--- apiVersion: apiserver.config.k8s.io/v1 kind: EncryptionConfiguration resources: @@ -127,33 +220,38 @@ resources: - aescbc: keys: - name: key1 + # See the following text for more details about the secret value secret: - - identity: {} + - identity: {} # this fallback allows reading unencrypted secrets; + # for example, during initial migratoin ``` -To create a new Secret, perform the following steps: +Place that base64 encoded value in the `secret` field of the encryption configuration file. -1. Generate a 32-byte random key and base64 encode it. If you're on Linux or macOS, run the following command: +### Reconfigure the API server - ```shell - head -c 32 /dev/urandom | base64 - ``` +{{< caution >}} +The encryption configuration file may contain keys that can decrypt content in etcd. +If the configuration file contains any key material, you must properly +restrict permissions on all your control plane hosts so only the user +who runs the kube-apiserver can read this configuration. + +Consider whether you need to make a backup of that configuration. If you do, +also think about how you will ensure that your backup is secure. +{{< /caution >}} -1. Place that value in the `secret` field of the `EncryptionConfiguration` struct. 1. Set the `--encryption-provider-config` flag on the `kube-apiserver` to point to - the location of the config file. -1. Restart your API server. + the location of the encryption configuration file. +1. Restart the API server. -{{< caution >}} -Your config file contains keys that can decrypt the contents in etcd, so you must properly restrict -permissions on your control-plane nodes so only the user who runs the `kube-apiserver` can read it. -{{< /caution >}} +If you have multiple API servers in your cluster, you should deploy the +changes in turn to each API server. -## Verifying that data is encrypted +### Verify that newly written data is encrypted {#verifying-that-data-is-encrypted} -Data is encrypted when written to etcd. After restarting your `kube-apiserver`, any newly created or -updated Secret should be encrypted when stored. To check this, you can use the `etcdctl` command line -program to retrieve the contents of your Secret. +Data is encrypted when written to etcd. After restarting your `kube-apiserver`, +any newly created or updated Secret should be encrypted when stored. To check this, +you can use the `etcdctl` command line program to retrieve the contents of your Secret. 1. Create a new Secret called `secret1` in the `default` namespace: @@ -161,7 +259,7 @@ program to retrieve the contents of your Secret. kubectl create secret generic secret1 -n default --from-literal=mykey=mydata ``` -1. Using the `etcdctl` command line, read that Secret out of etcd: +1. Using the `etcdctl` command line tool, read that Secret out of etcd: `ETCDCTL_API=3 etcdctl get /registry/secrets/default/secret1 [...] | hexdump -C` @@ -180,63 +278,112 @@ program to retrieve the contents of your Secret. [decoding a Secret](/docs/tasks/configmap-secret/managing-secret-using-kubectl/#decoding-secret) to completely decode the Secret. -## Ensure all Secrets are encrypted +### Ensure all Secrets are encrypted + +You have configured your cluster so that Secrets are encrypted on write. Performing +an update on Secret will encrypt that content at rest. -Since Secrets are encrypted on write, performing an update on a Secret will encrypt that content. +You can make this change across all Secrets in your cluster: ```shell +# Run this as an administrator that can read and write all Secrets kubectl get secrets --all-namespaces -o json | kubectl replace -f - ``` -The command above reads all Secrets and then updates them to apply server side encryption. +The command above reads all Secrets and then updates them with the same data, in order to +apply server side encryption. {{< note >}} If an error occurs due to a conflicting write, retry the command. -For larger clusters, you may wish to subdivide the secrets by namespace or script an update. +It is safe to run that command more than once. + +For larger clusters, you may wish to subdivide the Secrets by namespace, +or script an update. {{< /note >}} +## Cleanup {#cleanup-all-secrets-encrypted} + +You **must** ensure that all existing Secrets are encrypted at rest before +performing this cleanup step, because this change prevents the API server from +retrieving Secret resources (API path `secrets`) that are not encrypted. + +Once all Secrets in your cluster are encrypted, you can remove the `identity` +part of the encryption configuration. For example: + +```yaml +--- +apiVersion: apiserver.config.k8s.io/v1 +kind: EncryptionConfiguration +resources: + - resources: + - secrets + providers: + - aescbc: + keys: + - name: key1 + secret: + - identity: {} # REMOVE THIS LINE +``` + +…and then restart each API server in turn. This change prevents the API server +from accessing a plain-text Secret, even by accident. + ## Rotating a decryption key Changing a Secret without incurring downtime requires a multi-step operation, especially in the presence of a highly-available deployment where multiple `kube-apiserver` processes are running. -1. Generate a new key and add it as the second key entry for the current provider on all servers -1. Restart all `kube-apiserver` processes to ensure each server can decrypt using the new key -1. Make the new key the first entry in the `keys` array so that it is used for encryption in the config -1. Restart all `kube-apiserver` processes to ensure each server now encrypts using the new key -1. Run `kubectl get secrets --all-namespaces -o json | kubectl replace -f -` to encrypt all - existing Secrets with the new key -1. Remove the old decryption key from the config after you have backed up etcd with the new key in use - and updated all Secrets - -When running a single `kube-apiserver` instance, step 2 may be skipped. + + +1. Generate a new key and add it as the second key entry for the current provider on all + control plane hosts (strictly speaking: on all hosts that run a `kube-apiserver`). +2. Restart all `kube-apiserver` processes, to ensure each control plane host can decrypt + any data that are encrypted with the new key + (this is required for a replicated control plane; you can skip this step if you only run a + single `kube-apiserver`). +3. Make a secure backup of the new encryption key. If you lose all copies of this key you would + need to delete all the resources were encrypted under the lost key, and workloads may not + operate as expected during the time that at-rest encryption is broken. +4. Make the new key the first entry in the `keys` array so that it is used for encryption-at-rest + for new writes +5. Restart all `kube-apiserver` processes to ensure each control plane host now encrypts using the new key +6. As a privileged user, run `kubectl get secrets --all-namespaces -o json | kubectl replace -f -` + to encrypt all existing Secrets with the new key +7. After you have updated all existing Secrets to use the new key and have made a secure backup of the + new key, remove the old decryption key from the configuration. + +When your cluster only has a single `kube-apiserver` instance, you can skip step 2. ## Decrypting all data -To disable encryption at rest, place the `identity` provider as the first entry in the config -and restart all `kube-apiserver` processes. +To disable encryption at rest, place the `identity` provider as the first +entry your encryption configuration file: ```yaml +--- apiVersion: apiserver.config.k8s.io/v1 kind: EncryptionConfiguration resources: - resources: - secrets providers: - - identity: {} + - identity: {} # add this line - aescbc: keys: - name: key1 - secret: + secret: # keep this in place ``` -Then run the following command to force decrypt -all Secrets: +Then run the following command to force decryption of all Secrets: ```shell kubectl get secrets --all-namespaces -o json | kubectl replace -f - ``` +Once you have replaced all existing encrypted resources with backing data that +don't use encryption, you can remove the encryption settings from the +`kube-apiserver`. + ## {{% heading "whatsnext" %}} * Learn more about the [EncryptionConfiguration configuration API (v1)](/docs/reference/config-api/apiserver-encryption.v1/).