Skip to content

Commit

Permalink
Update encryption-at-rest task page
Browse files Browse the repository at this point in the history 
Co-authored-by: Divya Mohan <[email protected]>
  • Loading branch information
@sftim @divya-mohan0209
sftim and divya-mohan0209 committed May 21, 2022
1 parent 78a765f commit 98312e7
Showing 1 changed file with 140 additions and 47 deletions.
187 changes: 140 additions & 47 deletions content/en/docs/tasks/administer-cluster/encrypt-data.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -7,33 +7,56 @@ min-kubernetes-server-version: 1.13
---

<!-- overview -->
Some of the APIs in Kubernetes, such as {{< glossary_tooltip text="Secret" term_id="secret" >}},
support at-rest encryption. This at-rest encryption is additional to any system-level
encryption for the etcd cluster or hosts where the kube-apiserver stores data persistently.

This page shows how to enable and configure encryption of secret data at rest.

{{< note >}}
This task covers encryption for data stored using the
{{< glossary_tooltip text="Kubernetes API" term_id="kubernetes-api" >}}.

If you want to encrypt data in filesystems that are mounted into containers, you instead need
to use a storage integration that provides encrypted
{{< glossary_tooltip text="volumes" term_id="volume" >}}.
{{< /note >}}

## {{% heading "prerequisites" %}}

* {{< include "task-tutorial-prereqs.md" >}} {{< version-check >}}

* etcd v3.0 or later is required
* Your cluster's control plane must use etcd v3.x (major version 3, any minor version)

<!-- steps -->

## Configuration and determining whether encryption at rest is already enabled

By default, the API server uses `identity` provider is used to protect secrets in etcd, which
provides no encryption.

The `kube-apiserver` process accepts an argument `--encryption-provider-config`
that controls how API data is encrypted in etcd.
The configuration is provided as an API named
[`EncryptionConfiguration`](/docs/reference/config-api/apiserver-encryption.v1/).
An example configuration is provided below.
You can see an example configuration
in [Encryption at rest configuration](#understanding-the-encryption-at-rest-configuration).

{{< caution >}}
**IMPORTANT:** For high-availability configurations (with two or more control plane nodes), the
encryption configuration file must be the same! Otherwise, the `kube-apiserver` component cannot
decrypt data stored in the etcd.
For cluster configurations with two or more control plane nodes, the encryption configuration
must be identical across each control plane node.

If there is a difference in the encryption provider configuration, this may well mean
that the kube-apiserver can't decrypt data stored inside the key-value store (potentially
leading to further problems).
{{< /caution >}}

## Understanding the encryption at rest configuration.
## Encryption-at-rest configuration {#understanding-the-encryption-at-rest-configuration}

Here is an example EncryptionConfiguration file for the kube-apiserver:

```yaml
---
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
Expand All @@ -59,10 +82,15 @@ resources:
secret: YWJjZGVmZ2hpamtsbW5vcHFyc3R1dnd4eXoxMjM0NTY=
```
Each `resources` array item is a separate config and contains a complete configuration. The
`resources.resources` field is an array of Kubernetes resource names (`resource` or `resource.group`)
Each `resources` array item is a separate configuration and contains a complete configuration. The
`resources.resources` field is an array of Kubernetes resource types that the API server should
encrypt.
You can specify resource types as a simple type based on its URL representation (for example,
`secrets` for the [Secret API](/docs/reference/kubernetes-api/config-and-storage-resources/secret-v1/)).
You can also specify encryption for an API group (for example: `events.k8s.io`)
that should be encrypted. The `providers` array is an ordered list of the possible encryption
providers.
providers. Only one provider type may be specified per top-level entry
(for example: you can specify `identity` or `aescbc`, but not both in the same item).

Only one provider type may be specified per entry (`identity` or `aescbc` may be provided,
but not both in the same item).
Expand All @@ -71,28 +99,35 @@ resources from storage, each provider that matches the stored data attempts in o
data. If no provider can read the stored data due to a mismatch in format or secret key, an error
is returned which prevents clients from accessing that resource.

For more detailed information about the `EncryptionConfiguration` struct, please refer to the
[encryption configuration API](/docs/reference/config-api/apiserver-encryption.v1/).
For more detailed information about the `EncryptionConfiguration` for the kube-apiserver,
refer to the
[encryption configuration reference](/docs/reference/config-api/apiserver-encryption.v1/).

{{< caution >}}
If any resource is not readable via the encryption config (because keys were changed),
the only recourse is to delete that key from the underlying etcd directly. Calls that attempt to
read that resource will fail until it is deleted or a valid decryption key is provided.
{{< /caution >}}
{{< note >}}
If any resource is not readable via the encryption configuration (because keys were changed),
and you cannot restore a working configuration, your only recourse is to delete that key from
the underlying etcd directly. Calls that attempt to read that resource will fail until it is
deleted or a valid decryption key is provided.
{{< /note >}}

### Providers:
## Providers

{{< table caption="Providers for Kubernetes encryption at rest" >}}
Name | Encryption | Strength | Speed | Key Length | Other Considerations
Name | Encryption | Strength | Speed | Key length | Other considerations
-----|------------|----------|-------|------------|---------------------
`identity` | None | N/A | N/A | N/A | Resources written as-is without encryption. When set as the first provider, the resource will be decrypted as new values are written.
`secretbox` | XSalsa20 and Poly1305 | Strong | Faster | 32-byte | A newer standard and may not be considered acceptable in environments that require high levels of review.
`identity` | None | N/A | N/A | N/A | Resources written as-is without encryption. When set as the first provider, the resource will be decrypted as new values are written. Existing encrypted resource are **not** automatically overwritten with the plaintext data.
`aesgcm` | AES-GCM with random nonce | Must be rotated every 200k writes | Fastest | 16, 24, or 32-byte | Is not recommended for use except when an automated key rotation scheme is implemented.
`aescbc` | AES-CBC with PKCS#7 padding | Weak | Fast | 32-byte | Not recommended due to CBC's vulnerability to padding oracle attacks.
`kms` | Uses envelope encryption scheme: Data is encrypted by data encryption keys (DEKs) using AES-CBC with PKCS#7 padding, DEKs are encrypted by key encryption keys (KEKs) according to configuration in Key Management Service (KMS) | Strongest | Fast | 32-bytes | The recommended choice for using a third party tool for key management. Simplifies key rotation, with a new DEK generated for each encryption, and KEK rotation controlled by the user. [Configure the KMS provider](/docs/tasks/administer-cluster/kms-provider/)
`kms` | Uses envelope encryption scheme: Data is encrypted by data encryption keys (DEKs) using AES-CBC with PKCS#7 padding; DEKs are encrypted by key encryption keys (KEKs) according to configuration in Key Management Service (KMS) | Strongest | Fast | 32-bytes | The recommended choice for using a third party tool for key management. Simplifies key rotation, with a new DEK generated for each encryption, and KEK rotation controlled by the user. See [Configure the KMS provider](/docs/tasks/administer-cluster/kms-provider/). May incur extra costs for the external key management.
`secretbox` | XSalsa20 and Poly1305 | Strong | Faster | 32-byte | Uses relatively new encryption technologies that may not be considered acceptable in environments that require high levels of review.

You can configure multiple provides, and each provider (other than `identity`, which does not encrypt) supports multiple keys.
For encryption, the API server uses the first configured key from the first provider.
For decryption, the API server tries each key in order for decryption, stopping when decryption succeeds.
The API server can detect which provider to use for decryption based on metadata that accompanies the stored ciphertext.

Each provider supports multiple keys - the keys are tried in order for decryption, and if the provider
is the first provider, the first key is used for encryption.
Storing the raw encryption key in the EncryptionConfig only moderately improves your security posture, compared to no encryption.
Please use `kms` provider for additional security.


{{< caution >}}
Expand All @@ -115,9 +150,10 @@ retrieve the plaintext values, providing a higher level of security than locally

## Encrypting your data

Create a new encryption config file:
Create a new encryption configuration file. The contents should be similar to:

```yaml
---
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
Expand All @@ -127,41 +163,54 @@ resources:
- aescbc:
keys:
- name: key1
# See the following text for more details about this value.
secret: <BASE 64 ENCODED SECRET>
- identity: {}
- identity: {} # this fallback allows reading unencrypted secrets
```

To create a new Secret, perform the following steps:
To create a new encryption secret, perform the following steps:

1. Generate a 32-byte random key and base64 encode it. If you're on Linux or macOS, run the following command:

```shell
head -c 32 /dev/urandom | base64
```

1. Place that value in the `secret` field of the `EncryptionConfiguration` struct.
1. Place that value in the `secret` field of the encryption configuration file.
1. Set the `--encryption-provider-config` flag on the `kube-apiserver` to point to
the location of the config file.
1. Restart your API server.
1. Restart the API server.

If you have multiple API servers in your cluster, you should deploy the
changes in turn to each API server.


{{< note >}}
Keep the encryption secret confidential, including whilst you generate it and
ideally even after you are no longer actively using it.
{{< /note >}}

{{< caution >}}
Your config file contains keys that can decrypt the contents in etcd, so you must properly restrict
permissions on your control-plane nodes so only the user who runs the `kube-apiserver` can read it.
This configuration file contains keys that can decrypt content in etcd.
If the configuration file contains any key material, you must properly
restrict permissions on all your control plane hosts so only the user
who runs the kube-apiserver can read this configuration.
{{< /caution >}}


## Verifying that data is encrypted

Data is encrypted when written to etcd. After restarting your `kube-apiserver`, any newly created or
updated Secret should be encrypted when stored. To check this, you can use the `etcdctl` command line
program to retrieve the contents of your Secret.
Data is encrypted when written to etcd. After restarting your `kube-apiserver`,
any newly created or updated Secret should be encrypted when stored. To check this,
you can use the `etcdctl` command line program to retrieve the contents of your Secret.

1. Create a new Secret called `secret1` in the `default` namespace:

```shell
kubectl create secret generic secret1 -n default --from-literal=mykey=mydata
```

1. Using the `etcdctl` command line, read that Secret out of etcd:
1. Using the `etcdctl` command line tool, read that Secret out of etcd:

`ETCDCTL_API=3 etcdctl get /registry/secrets/default/secret1 [...] | hexdump -C`

Expand All @@ -182,41 +231,80 @@ program to retrieve the contents of your Secret.

## Ensure all Secrets are encrypted

Since Secrets are encrypted on write, performing an update on a Secret will encrypt that content.
You have configured your cluster so that Secrets are encrypted on write. Performing
an update on Secret will encrypt that content at rest.

You can make this change across all Secrets in your cluster:

```shell
# Run this as an administrator that can read and write all Secrets
kubectl get secrets --all-namespaces -o json | kubectl replace -f -
```

The command above reads all Secrets and then updates them to apply server side encryption.

{{< note >}}
If an error occurs due to a conflicting write, retry the command.
For larger clusters, you may wish to subdivide the secrets by namespace or script an update.
For larger clusters, you may wish to subdivide the Secrets by namespace, or script an update.
{{< /note >}}

### Cleanup {cleanup-all-secrets-encrypted}

Once all Secrets in your cluster are encrypted, you can remove the `identity`
part of the encryption configuration. For example:

```yaml
---
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
- secrets
providers:
- aescbc:
keys:
- name: key1
secret: <BASE 64 ENCODED SECRET>
- identity: {} # REMOVE THIS LINE
```

and then restart each API server in turn. This change prevents the API server
from accessing a plain-text Secret, so you **must** first ensure that there are
no resources of that type (`secrets`, for this example) before performing the
clean up steps.

## Rotating a decryption key

Changing a Secret without incurring downtime requires a multi-step operation, especially in
the presence of a highly-available deployment where multiple `kube-apiserver` processes are running.

1. Generate a new key and add it as the second key entry for the current provider on all servers
1. Restart all `kube-apiserver` processes to ensure each server can decrypt using the new key
1. Make the new key the first entry in the `keys` array so that it is used for encryption in the config
1. Restart all `kube-apiserver` processes to ensure each server now encrypts using the new key
1. Run `kubectl get secrets --all-namespaces -o json | kubectl replace -f -` to encrypt all
existing Secrets with the new key
1. Remove the old decryption key from the config after you have backed up etcd with the new key in use
and updated all Secrets
<!-- manual numbering use to help locate “step 2” for following text -->
1. Generate a new key and add it as the second key entry for the current provider on all
control plane hosts (strictly speaking: on all hosts that run a kube-apiserver).
2. Restart all `kube-apiserver` processes, to ensure each control plane host can decrypt
any data that are encrypted with the new key
(this is required for a replicated control plane; you can skip this step if you only run a
single `kube-apiserver`).
3. Make a secure backup of the new encryption key. If you lose all copies of this key you would
need to delete all the resources were encrypted under the lost key, and workloads may not
operate as expected during the time that at-rest encryption is broken.
4. Make the new key the first entry in the `keys` array so that it is used for encryption-at-rest
for new writes
5. Restart all `kube-apiserver` processes to ensure each control plane host now encrypts using the new key
6. As a privileged user, run `kubectl get secrets --all-namespaces -o json | kubectl replace -f -`
to encrypt all existing Secrets with the new key
7. After you have updated all existing Secrets to use the new key and have made a secure backup of the
new key, remove the old decryption key from the configuration.

When running a single `kube-apiserver` instance, step 2 may be skipped.

## Decrypting all data

To disable encryption at rest, place the `identity` provider as the first entry in the config
and restart all `kube-apiserver` processes.
To disable encryption at rest, place the `identity` provider as the first
entry your encryption configuration file:

```yaml
---
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
Expand All @@ -227,7 +315,7 @@ resources:
- aescbc:
keys:
- name: key1
secret: <BASE 64 ENCODED SECRET>
secret: <BASE 64 ENCODED SECRET> # keep this in place
```

Then run the following command to force decrypt
Expand All @@ -236,6 +324,11 @@ all Secrets:
```shell
kubectl get secrets --all-namespaces -o json | kubectl replace -f -
```
to force all Secrets to be decrypted.

Once you have replaced all existing encrypted resources with backing data that
don't use encryption, you can remove the encryption settings from the
kube-apiserver.

## {{% heading "whatsnext" %}}

Expand Down

0 comments on commit 98312e7

Please sign in to comment.