Doc 2599 - Configure Redpanda with a Customer-Managed Key

modules/manage/partials/tiered-storage.adoc

@@ -58,6 +58,11 @@ TIP: If deploying Redpanda on an AWS Auto-Scaling group (ASG), keep in mind that
 
 You can configure access to Amazon S3 with either an IAM role attached to the instance or with access keys.
 
+[TIP]
+====
+If you need to manage and store encryption keys separately from your cloud provider, you can <<configure-access-to-aws-s3-using-an-aws-kms-key,configure access to an AWS S3 bucket that Redpanda Tiered Storage uses to leverage your AWS KMS key (SSE-KMS)>> instead of the default AWS S3-managed key (SSE-S3). This option enables you to segregate data from different teams or departments and remove that data at will by removing the encryption keys.
+====
+
 ===== Use IAM roles
 
 To configure access to an S3 bucket with an IAM role:
@@ -258,9 +263,11 @@ Replace the following placeholders:
 +
 CAUTION: Do not set an object storage property to an empty string `""` or to `null` as a way to reset it to its default value.
 
+include::manage:partial$tiered-storage/aws-kms-key.adoc[leveloffset=+4]
+
 ==== Google Cloud Storage
 
-You can configure access to Google Cloud Storage with either an IAM role attached to the instance or with access keys.
+Configure access to Google Cloud Storage with either an IAM role attached to the instance, with access keys, or customer-managed keys.
 
 ===== Use IAM roles
 
@@ -462,6 +469,8 @@ Replace the following placeholders:
 +
 CAUTION: Do not set an object storage property to an empty string `""` or to `null` as a way to reset it to its default value.
 
+include::manage:partial$tiered-storage/gcp-kms-key.adoc[leveloffset=+4]
+
 ==== Microsoft ABS/ADLS
 
 You can configure access to Azure Blob Storage with either account access keys or Azure's managed identities system to securely interact with Azure Blob Storage. Account access keys, as static credentials, require manual management and vigilant security practices to prevent breaches due to their unchanging nature. In contrast, managed identities provide a more secure and maintenance-free solution by automating credential management and rotation, though they are exclusive to the Azure ecosystem.
@@ -688,7 +697,12 @@ Amazon S3::
 
 TIP: If deploying Redpanda on an AWS Auto-Scaling group (ASG), keep in mind that the ASG controller terminates nodes and spins up replacements if the nodes saturate and are unable to heartbeat the controller (based on the EC2 health check). For more information, see the https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/monitoring-system-instance-status-check.html#types-of-instance-status-checks[AWS documentation^]. Redpanda recommends deploying on Linux or Kubernetes. For more information, see xref:deploy:deployment-option/self-hosted/index.adoc[Deploy Redpanda].
 
-Configure access to Amazon S3 with either an IAM role attached to the instance or with access keys.
+Configure access to Amazon S3 with either an IAM role attached to the instance, or with access keys.
+
+[TIP]
+====
+If you need to manage and store encryption keys separately from your cloud provider, you can <<configure-access-to-aws-s3-using-an-aws-kms-key,configure access to an AWS S3 bucket that Redpanda Tiered Storage uses to leverage your AWS KMS key (SSE-KMS)>> instead of the default AWS S3-managed key (SSE-S3). This option enables you to segregate data from different teams or departments and remove that data at will by removing the encryption keys.
+====
 
 To configure access to an S3 bucket with an IAM role:
 
@@ -726,11 +740,13 @@ Replace `<placeholders>` with your own values.
 +
 CAUTION: Do not set an object storage property to an empty string `""` or to `null` as a way to reset it to its default value. To reset a property to its default value, run `rpk cluster config force-reset <config-name>` or remove that line from the cluster configuration with `rpk cluster config edit`.
 
+include::manage:partial$tiered-storage/aws-kms-key.adoc[leveloffset=+3]
+
 --
 Google Cloud Storage::
 +
 --
-Configure access to Google Cloud Storage with either an IAM role attached to the instance or with access keys.
+Configure access to Google Cloud Storage with either an IAM role attached to the instance, with access keys, or customer-managed keys.
 
 To configure access to Google Cloud Storage with an IAM role:
 
@@ -772,6 +788,8 @@ Replace `<placeholders>` with your own values.
 +
 CAUTION: Do not set an object storage property to an empty string `""` or to `null` as a way to reset it to its default value. To reset a property to its default value, run `rpk cluster config force-reset <config-name>` or remove that line from the cluster configuration with `rpk cluster config edit`.
 
+include::manage:partial$tiered-storage/gcp-kms-key.adoc[leveloffset=+3]
+
 --
 Microsoft ABS/ADLS::
 +

modules/manage/partials/tiered-storage/aws-kms-key.adoc

@@ -0,0 +1,67 @@
+[discrete]
+= Configure access to AWS S3 using an AWS KMS key
+
+When there are strict data compliance requirements and you must manage and store encryption keys separately from your cloud provider, you can configure an Amazon S3 bucket that Tiered Storage can use to leverage your customer-provided key (SSE-KMS) instead of the default AWS-managed key (SSE-S3).
+
+To convert an existing S3 bucket and its contents, you must:
+
+. Create a new KMS key
+. Configure the S3 bucket to use the new KMS key
+. (Optional) Re-encrypt existing objects to use the new KMS key
+
+[NOTE]
+====
+ifdef::env-cloud[]
+You cannot configure a cloud-provider managed encryption key at the topic level or in Redpanda Cloud Dedicated clusters.
+endif::[]
+ifndef::env-cloud[]
+You cannot configure a cloud-provider managed encryption key at the topic level.
+endif::[]
+
+For topic-level control, each CLI Get or Put for a partition must use the correct key as configured on the topic.
+====
+
+[discrete]
+== Prerequisites
+
+- The user configuring S3 bucket encryption must be assigned Key admin permission. Without this permission, you will be unable to re-encrypt existing bucket objects to use the KMS key.
+- The S3 bucket must be assigned Key user permission. Without this permission, Redpanda will be unable to write new objects to Tiered Storage.
+- If you intend to retroactively re-encrypt existing data with the new KMS key, record the ARN identifier of the key upon creation, as it will be required later when running an AWS CLI command.
+
+To create a new KMS key in the AWS Console:
+
+. In AWS Console, search for “Key Management Service”.
+. Click **Create a key**.
+. On the Configure key page, select the Symmetric key type, then select Encrypt and decrypt.
+. Click the **Advanced options** tab and configure Key material origin and Regionality as needed. For example, if you are using xref:manage:remote-read-replicas.adoc[Remote Read Replicas] or have Redpanda spanning across regions, select Multi-Region key.
+. Click **Next**.
+. On the Add labels page, specify an alias name and description for the key. Do not include sensitive information in these fields.
+. Click **Next**.
+. On the Define key administrative permissions page, specify a user who can administer this key through the KMS API.
+. Click **Next**.
+. On the Define key usage permissions page, assign the S3 bucket as a Key user. This is required for the S3 bucket to encrypt and decrypt.
+. Click **Next**.
+. Review your KMS key configuration and click **Finish**.
+
+For more information, see the AWS documentation https://docs.aws.amazon.com/kms/latest/developerguide/create-symmetric-cmk.html[Creating a symmetric encryption KMS key^].
+
+To configure the S3 bucket to use the new KMS key (and reduce KMS costs through caching):
+
+. In AWS Console, search for "S3".
+. Select the bucket used by Redpanda.
+. Click the **Properties** tab.
+. In Default encryption, click **Edit**.
+. For Encryption type, select “Server-side encryption with AWS Key Management Service keys (SSE-KMS)”.
+. Locate and select your AWS KMS key ARN identifier.
+. Click **Save changes**.
+
+(Optional) To re-encrypt existing data using the new KMS key:
+
+Existing data in your S3 bucket continues to be read using the AWS-managed key, while moving forward, new objects are encrypted using the new KMS key. If you wish to re-encrypt all S3 bucket data to use the KMS key, run the following AWS CLI command:
+
+[,bash]
+----
+aws s3 cp s3://{BUCKET_NAME}/ s3://{BUCKET_NAME}/ --recursive --sse-kms-key-id {KMS_KEY_ARN} --sse aws:kms
+----
+
+For more information, see the AWS documentation https://docs.aws.amazon.com/AmazonS3/latest/userguide/configuring-bucket-key.html[Enable an S3 Bucket Key for an existing bucket^].

modules/manage/partials/tiered-storage/gcp-kms-key.adoc

@@ -0,0 +1,68 @@
+[discrete]
+= Configure access to data in GCP using a KMS key
+
+To configure the Google Cloud bucket used by Redpanda Tiered Storage to leverage a customer-managed key using the Cloud Key Management Service API instead of the default Google-managed key, you must:
+. Create a KMS key.
+. Configure the bucket to use the KMS key.
+. Optionally, re-encrypt existing data with the new KMS key.
+To manage Google Cloud KMS using the command line, first install or upgrade to the latest version of https://cloud.google.com/sdk/docs/install[Google Cloud CLI^].
+To create a KMS key:
+. In Google Cloud Console, search for "Cloud Key Managment Service API". Click **Enable** if the service is not already enabled.
+. Using the Google Cloud CLI, create a new keyring in the https://cloud.google.com/kms/docs/locations^[region] where the bucket used by Redpanda is located. Note that region is case-sensitive.
++
+[,bash, indent]
+----
+gcloud kms keyrings create "redpanda-keyring" --location="{REGION}"
+----
++
+. Create a new key for the keyring in the same region as the bucket:
++
+[,bash, indent]
+----
+gcloud kms keys create "redpanda-key" \
+  --location="{REGION}" \
+  --keyring="redpanda-keyring" \
+  --purpose="encryption"
+----
++
+. Get the key identifier:
++
+[,bash]
+----
+gcloud kms keys list \
+  --location="REGION" \
+  --keyring="redpanda-keyring"
+----
++
+The result should look like the following. Be sure to record the name, as this is used to assign and manage the key, and use this as the \{KEY_RESOURCE} placeholder in subsequent commands.
++
+[,bash]
+----
+NAME      PURPOSE      ALGORITHM      PROTECTION_LEVEL    LABELS    PRIMARY_ID    PRIMARY_STATE
+projects/{PROJECT_NAME}/locations/us/keyRings/redpanda-keyring/cryptoKeys/redpanda-key
+ENCRYPT_DECRYPT  GOOGLE_SYMMETRIC_ENCRYPTION  SOFTWARE          1           ENABLED
+----
+To configure the GCP bucket to use the KMS key:
+. Assign the key to a service agent:
++
+[,bash]
+----
+gcloud storage service-agent \
+  --project={PROJECT_ID_STORING_OBJECTS} \
+  --authorize-cmek={KEY_RESOURCE}
+----
++
+. Set the bucket default encryption key to the KMS key:
++
+[,bash]
+----
+gcloud storage buckets update gs://{BUCKET_NAME} \
+  --default-encryption-key={KEY_RESOURCE}
+----
+(Optional) To re-encrypt existing data using the new KMS key:
+Existing data in the bucket will continue to be read using the Google-managed key, while moving forward, new objects are encrypted using the new KMS key. If you wish to re-encrypt all data in the bucket to use the KMS key, run the following command:
+[,bash]
+----
+gcloud storage objects update gs://{BUCKET_NAME}/ --recursive \
+  --encryption-key={KEY_RESOURCE}
+----