-
Notifications
You must be signed in to change notification settings - Fork 34
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Doc 2599 - Configure Redpanda with a Customer-Managed Key (#808)
Co-authored-by: JakeSCahill <[email protected]> Co-authored-by: Michele Cyran <[email protected]> Co-authored-by: Jake Cahill <[email protected]>
- Loading branch information
4 people
authored
Nov 15, 2024
1 parent
ddbe343
commit 183d021
Showing
4 changed files
with
183 additions
and
27 deletions.
There are no files selected for viewing
5 changes: 2 additions & 3 deletions
5
modules/manage/pages/kubernetes/tiered-storage/k-tiered-storage.adoc
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,10 +1,9 @@ | ||
| = Use Tiered Storage in Kubernetes | ||
| :description: Configure your Redpanda cluster to offload log segments to object storage and save storage costs. | ||
| :page-context-links: [{"name": "Linux", "to": "manage:tiered-storage.adoc" },{"name": "Kubernetes", "to": "manage:kubernetes/storage/tiered-storage/k-tiered-storage.adoc" } ] | ||
| :page-categories: Management, High Availability, Data Replication | ||
| :env-kubernetes: true | ||
| :tags: ["Kubernetes", "Helm configuration"] | ||
| :page-toclevels: 4 | ||
| :page-aliases: manage:kubernetes/tiered-storage.adoc, manage:kubernetes/storage/tiered-storage.adoc, manage:kubernetes/data-archiving.adoc, manage:kubernetes/storage/data-archiving.adoc, manage:kubernetes/storage/tiered-storage/k-data-archiving.adoc, manage:kubernetes/storage/tiered-storage/k-tiered-storage.adoc | ||
|
|
||
|
|
||
| include::manage:partial$tiered-storage.adoc[] | ||
| include::manage:partial$tiered-storage.adoc[] |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,71 @@ | ||
| ifndef::env-kubernetes[] | ||
| [discrete] | ||
| endif::[] | ||
| = **Configure access with an AWS KMS key** | ||
|
|
||
| When there are strict data compliance requirements and you must manage and store encryption keys separate from your cloud provider, you can configure an Amazon S3 bucket that Tiered Storage can use to leverage your customer-provided key (SSE-KMS) instead of the default AWS-managed key (SSE-S3). | ||
|
|
||
| To convert an existing S3 bucket and its contents, you must: | ||
|
|
||
| . Create a new KMS key. | ||
| . Configure the S3 bucket to use the new KMS key. | ||
| . (Optional) Re-encrypt existing objects to use the new KMS key. | ||
|
|
||
| [NOTE] | ||
| ==== | ||
| ifdef::env-cloud[] | ||
| You cannot configure a cloud-provider managed encryption key at the topic level or in Redpanda Cloud Dedicated clusters. | ||
| endif::[] | ||
| ifndef::env-cloud[] | ||
| You cannot configure a cloud provider-managed encryption key at the topic level. | ||
| endif::[] | ||
| For topic-level control, each CLI Get or Put for a partition must use the correct key as configured on the topic. | ||
| ==== | ||
|
|
||
| ifndef::env-kubernetes[] | ||
| [discrete] | ||
| endif::[] | ||
| == **Prerequisites** | ||
|
|
||
| - The user configuring S3 bucket encryption must be assigned the Key admin permission. Without this permission, the user is unable to re-encrypt existing bucket objects to use the KMS key. | ||
| - The S3 bucket must be assigned the Key user permission. Without this permission, Redpanda is unable to write new objects to Tiered Storage. | ||
| - If you intend to retroactively re-encrypt existing data with the new KMS key, store the ARN identifier of the key upon creation. It is required for AWS CLI commands. | ||
| To create a new KMS key in the AWS Console: | ||
|
|
||
| . In AWS Console, search for “Key Management Service”. | ||
| . Click **Create a key**. | ||
| . On the Configure key page, select the **Symmetric** key type, then select **Encrypt and decrypt**. | ||
| . Click the **Advanced options** tab and configure Key material origin and Regionality as needed. For example, if you are using xref:manage:remote-read-replicas.adoc[Remote Read Replicas] or have Redpanda spanning across regions, select **Multi-Region key**. | ||
| . Click **Next**. | ||
| . On the Add labels page, specify an alias name and description for the key. Do not include sensitive information in these fields. | ||
| . Click **Next**. | ||
| . On the Define key administrative permissions page, specify a user who can administer this key through the KMS API. | ||
| . Click **Next**. | ||
| . On the Define key usage permissions page, assign the S3 bucket as a Key user. This is required for the S3 bucket to encrypt and decrypt. | ||
| . Click **Next**. | ||
| . Review your KMS key configuration and click **Finish**. | ||
|
|
||
| For more information, see the https://docs.aws.amazon.com/kms/latest/developerguide/create-symmetric-cmk.html[AWS documentation^]. | ||
|
|
||
| To configure the S3 bucket to use the new KMS key (and reduce KMS costs through caching): | ||
|
|
||
| . In AWS Console, search for "S3". | ||
| . Select the bucket used by Redpanda. | ||
| . Click the **Properties** tab. | ||
| . In Default encryption, click **Edit**. | ||
| . For Encryption type, select “Server-side encryption with AWS Key Management Service keys (SSE-KMS)”. | ||
| . Locate and select your AWS KMS key ARN identifier. | ||
| . Click **Save changes**. | ||
|
|
||
| (Optional) To re-encrypt existing data using the new KMS key: | ||
|
|
||
| Existing data in your S3 bucket continues to be read using the AWS-managed key, while new objects are encrypted using the new KMS key. If you wish to re-encrypt all S3 bucket data to use the KMS key, run: | ||
|
|
||
| [,bash] | ||
| ---- | ||
| aws s3 cp s3://{BUCKET_NAME}/ s3://{BUCKET_NAME}/ --recursive --sse-kms-key-id {KMS_KEY_ARN} --sse aws:kms | ||
| ---- | ||
|
|
||
| For more information, see the https://docs.aws.amazon.com/AmazonS3/latest/userguide/configuring-bucket-key.html[AWS documentation^]. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,78 @@ | ||
| ifndef::env-kubernetes[] | ||
| [discrete] | ||
| endif::[] | ||
| = **Configure access with a KMS key** | ||
|
|
||
| To configure the Google Cloud bucket used by Redpanda Tiered Storage to leverage a customer-managed key using the Cloud Key Management Service API instead of the default Google-managed key, you must: | ||
|
|
||
| . Create a KMS key. | ||
| . Configure the bucket to use the KMS key. | ||
| . Optionally, re-encrypt existing data with the new KMS key. | ||
|
|
||
| To manage Google Cloud KMS using the command line, first install or upgrade to the latest version of https://cloud.google.com/sdk/docs/install[Google Cloud CLI^]. | ||
|
|
||
| To create a KMS key: | ||
|
|
||
| . In Google Cloud Console, search for "Cloud Key Managment Service API". Click **Enable** if the service is not already enabled. | ||
| . Using the Google Cloud CLI, create a new keyring in the https://cloud.google.com/kms/docs/locations^[region] where the bucket used by Redpanda is located. Note that region is case sensitive. | ||
| + | ||
| [,bash, indent] | ||
| ---- | ||
| gcloud kms keyrings create "redpanda-keyring" --location="{REGION}" | ||
| ---- | ||
| + | ||
| . Create a new key for the keyring in the same region as the bucket: | ||
| + | ||
| [,bash, indent] | ||
| ---- | ||
| gcloud kms keys create "redpanda-key" \ | ||
| --location="{REGION}" \ | ||
| --keyring="redpanda-keyring" \ | ||
| --purpose="encryption" | ||
| ---- | ||
| + | ||
| . Get the key identifier: | ||
| + | ||
| [,bash] | ||
| ---- | ||
| gcloud kms keys list \ | ||
| --location="REGION" \ | ||
| --keyring="redpanda-keyring" | ||
| ---- | ||
| + | ||
| The result should look like the following. Be sure to store the name, as this is used to assign and manage the key. Use this as the \{KEY_RESOURCE} placeholder in subsequent commands. | ||
| + | ||
| [,bash] | ||
| ---- | ||
| NAME PURPOSE ALGORITHM PROTECTION_LEVEL LABELS PRIMARY_ID PRIMARY_STATE | ||
| projects/{PROJECT_NAME}/locations/us/keyRings/redpanda-keyring/cryptoKeys/redpanda-key | ||
| ENCRYPT_DECRYPT GOOGLE_SYMMETRIC_ENCRYPTION SOFTWARE 1 ENABLED | ||
| ---- | ||
|
|
||
| To configure the GCP bucket to use the KMS key: | ||
|
|
||
| . Assign the key to a service agent: | ||
| + | ||
| [,bash] | ||
| ---- | ||
| gcloud storage service-agent \ | ||
| --project={PROJECT_ID_STORING_OBJECTS} \ | ||
| --authorize-cmek={KEY_RESOURCE} | ||
| ---- | ||
| + | ||
| . Set the bucket default encryption key to the KMS key: | ||
| + | ||
| [,bash] | ||
| ---- | ||
| gcloud storage buckets update gs://{BUCKET_NAME} \ | ||
| --default-encryption-key={KEY_RESOURCE} | ||
| ---- | ||
|
|
||
| (Optional) To re-encrypt existing data using the new KMS key: | ||
|
|
||
| Existing data in the bucket continues to be read using the Google-managed key, while new objects are encrypted using the new KMS key. If you wish to re-encrypt all data in the bucket to use the KMS key, run: | ||
| [,bash] | ||
| ---- | ||
| gcloud storage objects update gs://{BUCKET_NAME}/ --recursive \ | ||
| --encryption-key={KEY_RESOURCE} | ||
| ---- |