Add docs for extraClusterConfig (#1187)

Co-authored-by: Chris Seto <[email protected]>
Co-authored-by: Michele Cyran <[email protected]>

Author: 3 people

modules/get-started/pages/release-notes/helm-charts.adoc

@@ -44,6 +44,20 @@ The connectors subchart has been removed from the Helm chart because it is not o
 
 TIP: For a supported and scalable Kafka Connect alternative, consider using Redpanda Connect. For more information, see xref:redpanda-connect:get-started:index.adoc[].
 
+=== Reference Kubernetes Secrets and ConfigMaps for Redpanda cluster configuration
+
+You can now set any Redpanda cluster configuration property using the new `extraClusterConfig` field. This allows you to reference values from Kubernetes Secrets or ConfigMaps. For example, use this field to inject sensitive credentials or reuse shared configurations across features like Tiered Storage, Iceberg, and disaster recovery.
+
+This enhancement improves:
+
+- Security: Avoid hardcoding secrets in Helm values or manifests.
+
+- Reusability: Centralize common values used by multiple features.
+
+- Maintainability: Better integrate with GitOps workflows and Kubernetes-native resource management.
+
+See xref:manage:kubernetes/k-configure-helm-chart.adoc#extra-cluster-config[Set Redpanda cluster properties from Kubernetes Secrets or ConfigMaps].
+
 == Redpanda chart v5.10.x
 
 === Ability to change StatefulSet replicas without restarting brokers

modules/get-started/pages/release-notes/operator.adoc

@@ -72,6 +72,20 @@ The connectors subchart has been removed from the Helm chart because it is not o
 
 TIP: For a supported and scalable Kafka Connect alternative, consider using Redpanda Connect. For more information, see xref:redpanda-connect:get-started:index.adoc[].
 
+=== Reference Kubernetes Secrets and ConfigMaps for Redpanda cluster configuration
+
+You can now set any Redpanda cluster configuration property using the new `extraClusterConfig` field. This allows you to reference values from Kubernetes Secrets or ConfigMaps. For example, use this field to inject sensitive credentials or reuse shared configurations across features like Tiered Storage, Iceberg, and disaster recovery.
+
+This enhancement improves:
+
+- Security: Avoid hardcoding secrets in Helm values or manifests.
+
+- Reusability: Centralize common values used by multiple features.
+
+- Maintainability: Better integrate with GitOps workflows and Kubernetes-native resource management.
+
+See xref:manage:kubernetes/k-configure-helm-chart.adoc#extra-cluster-config[Set Redpanda cluster properties from Kubernetes Secrets or ConfigMaps].
+
 == Redpanda Operator v2.4.x
 
 link:https://github.com/redpanda-data/redpanda-operator/blob/release/v2.4.x/operator/CHANGELOG.md[Changelog^].

modules/manage/pages/kubernetes/k-configure-helm-chart.adoc

@@ -210,6 +210,147 @@ helm upgrade --install redpanda redpanda/redpanda \
 --
 ======
 
+[[extra-cluster-config]]
+== Set Redpanda cluster properties from Kubernetes Secrets or ConfigMaps
+
+Starting in v25.1.1 of the Redpanda Operator and Redpanda Helm chart, you can set **any Redpanda cluster configuration property** by referencing Kubernetes Secrets or ConfigMaps using the `config.extraClusterConfig` field.
+
+This feature provides a more secure, maintainable, and declarative way to manage sensitive or shared configuration values across your Redpanda deployment.
+
+Use this method to:
+
+- Securely inject sensitive values, such as credentials for Iceberg, TLS, or object storage.
+- Reuse the same value across multiple features, such as Tiered Storage, Iceberg, and disaster recovery, without duplication.
+- Centralize config management in Kubernetes-native resources to support GitOps and reduce drift.
+
+For example, to set `iceberg_rest_catalog_client_secret` using a Secret called `iceberg-config`:
+
+[tabs]
+======
+Operator::
++
+--
+.`redpanda-cluster.yaml`
+[,yaml]
+----
+apiVersion: cluster.redpanda.com/v1alpha2
+kind: Redpanda
+metadata:
+  name: redpanda
+spec:
+  clusterSpec:
+    config:
+      extraClusterConfig:
+        iceberg_rest_catalog_client_secret:
+          secretRef:
+            name: iceberg-config
+            key: iceberg_rest_catalog_client_secret
+----
+
+```bash
+kubectl apply -f redpanda-cluster.yaml --namespace <namespace>
+```
+
+--
+Helm::
++
+--
+[tabs]
+====
+--values::
++
+.`redpanda-config.yaml`
+[,yaml]
+----
+  config:
+    extraClusterConfig:
+      iceberg_rest_catalog_client_secret:
+        secretRef:
+          name: iceberg-config
+          key: iceberg_rest_catalog_client_secret
+----
++
+```bash
+helm upgrade --install redpanda redpanda/redpanda \
+    --namespace <namespace> --create-namespace \
+    --values redpanda-configs.yaml --reuse-values
+```
+
+--set::
++
+[,bash,role="no-wrap"]
+----
+helm upgrade --install redpanda redpanda/redpanda \
+  --namespace <namespace> \
+  --create-namespace \
+  --set config.extraClusterConfig.iceberg_rest_catalog_client_secret.secretRef.name=iceberg-config \
+  --set config.extraClusterConfig.iceberg_rest_catalog_client_secret.secretRef.key=iceberg_rest_catalog_client_secret
+----
+
+====
+--
+======
+
+This method supports both `secretRef` and `configMapRef`:
+
+- Use `secretRef` for sensitive data like access keys or credentials.
+
+- Use `configMapRef` for shared, non-sensitive values such as URIs or feature flags.
+
+You can apply this approach to any Redpanda configuration key, making your deployments more secure, modular, and easier to manage at scale.
+
+For full configuration options, see xref:reference:properties/index.adoc[].
+
+[[reset-config]]
+== Reset configuration values
+
+You may want to reset a configuration value back to its default. The method to do this depends on how you're managing your Redpanda deployment.
+
+[tabs]
+======
+Operator::
++
+--
+
+If you're using the Redpanda Operator and want to reset a configuration property back to its default:
+
+. Add the following annotation to your Redpanda custom resource to enable declarative configuration sync:
++
+[source,yaml]
+----
+metadata:
+  annotations:
+    operator.redpanda.com/config-sync-mode: Declarative
+----
+
+. Remove the configuration key you want to reset from `spec.clusterSpec.config`.
+
+With this annotation, the Redpanda Operator ensures that removed keys are also removed from the Redpanda cluster configuration.
+
+
+If this annotation is not set, the Redpanda Operator retains previously applied values even if you remove them from the custom resource.
+
+--
+CLI::
++
+--
+
+To reset a configuration property using the Redpanda CLI:
+
+- Run the xref:reference:rpk/rpk-cluster/rpk-cluster-config-set.adoc[`rpk cluster config set`] command with an empty string:
+
+[source,bash]
+----
+rpk cluster config set <property> ""
+----
+
+- Or, use the xref:reference:rpk/rpk-cluster/rpk-cluster-config-edit.adoc[`rpk cluster config edit`] command and delete the line for the property.
+
+If you're using a file, such as a `values.yaml` or a Redpanda resource, to manage your configuration, make sure to also remove the property from that file. Otherwise, it may be reapplied the next time you run `helm upgrade` or the Pods restart.
+
+--
+======
+
 == Configure Redpanda Console
 
 Redpanda Console is included as a subchart of the Redpanda Helm chart.

modules/manage/partials/kubernetes/extraclusterconfig.adoc

@@ -0,0 +1,6 @@
+[TIP]
+====
+Starting in Redpanda Operator v25.1.1, you can configure object storage settings using `extraClusterConfig`. This lets you securely reference sensitive values from Kubernetes Secrets or ConfigMaps, and reuse values like your bucket name across multiple features, such as Tiered Storage, Iceberg, and topic recovery.
+
+See xref:manage:kubernetes/k-configure-helm-chart.adoc#extra-cluster-config[Set Redpanda cluster properties from Kubernetes Secrets or ConfigMaps].
+====

modules/manage/partials/kubernetes/reset-object-storage-properties.adoc

@@ -0,0 +1,2 @@
+CAUTION: Do not set an object storage property to an empty string (`""`) or `null` to reset it. This may result in invalid or incomplete configuration. For safe ways to reset a property, see xref:manage:kubernetes/k-configure-helm-chart.adoc#reset-config[Reset configuration values].
+

modules/manage/partials/tiered-storage.adoc

@@ -33,7 +33,7 @@ rpk cluster license info
 - Migrating topics from one bucket or container to another is not supported.
 - Multi-region buckets or containers are not supported.
 
-CAUTION: Redpanda strongly recommends that you do not re-enable Tiered Storage after previously enabling and disabling it. Re-enabling Tiered Storage can result in inconsistent data and data gaps in Tiered Storage for a topic.
+CAUTION: Redpanda Data recommends that you do not re-enable Tiered Storage after previously enabling and disabling it. Re-enabling Tiered Storage can result in inconsistent data and data gaps in Tiered Storage for a topic.
 
 == Set up Tiered Storage
 
@@ -50,14 +50,18 @@ Redpanda natively supports Tiered Storage with Amazon Simple Storage Service (S3
 ifdef::env-kubernetes[]
 ==== Amazon S3
 
-TIP: If deploying Redpanda on an AWS Auto-Scaling group (ASG), keep in mind that the ASG controller terminates nodes and spins up replacements if the nodes saturate and are unable to heartbeat the controller (based on the EC2 health check). For more information, see the https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/monitoring-system-instance-status-check.html#types-of-instance-status-checks[AWS documentation^]. Redpanda recommends deploying on Linux or Kubernetes. For more information, see xref:deploy:deployment-option/self-hosted/index.adoc[Deploy Redpanda].
+[NOTE]
+====
+If you deploy Redpanda directly on AWS EC2 instances managed by an Auto-Scaling Group (ASG), be aware that ASG may terminate and replace instances based on system-level health checks. This can result in unexpected Redpanda broker terminations, risking availability or data loss.
 
-You can configure access to Amazon S3 with either an IAM role attached to the instance or with access keys.
+Redpanda Data recommends deploying on Kubernetes or using other orchestrators that understand Redpanda's stateful nature and can handle Pod lifecycle and storage gracefully.
 
-[TIP]
+See xref:deploy:deployment-option/self-hosted/index.adoc[Deploy Redpanda] for deployment best practices.
 ====
+
+You can configure access to Amazon S3 with either an IAM role attached to the instance or with access keys.
+
 If you need to manage and store encryption keys separately from your cloud provider, you can <<configure-access-with-an-aws-kms-key,configure access to an AWS S3 bucket that Redpanda Tiered Storage uses to leverage your AWS KMS key (SSE-KMS)>> instead of the default AWS S3-managed key (SSE-S3). This option enables you to segregate data from different teams or departments and remove that data at will by removing the encryption keys.
-====
 
 ===== **Configure access with an IAM role**
 
@@ -135,11 +139,13 @@ Replace the following placeholders:
 - `<region>`: The region of your S3 bucket.
 - `<redpanda-bucket-name>`: The name of your S3 bucket.
 +
-CAUTION: Do not set an object storage property to an empty string `""` or to `null` as a way to reset it to its default value.
+include::manage:partial$kubernetes/reset-object-storage-properties.adoc[]
+
+include::manage:partial$kubernetes/extraclusterconfig.adoc[]
 
 ===== **Configure access with access keys**
 
-. Grant an IAM user the following permissions to read and create objects in your buckets:
+. In AWS, grant an IAM user the following permissions to read and create objects in your buckets:
 - `GetObject`
 - `DeleteObject`
 - `PutObject`
@@ -253,7 +259,9 @@ Replace the following placeholders:
 - `<region>`: The region of your S3 bucket.
 - `<redpanda-bucket-name>`: The name of your S3 bucket.
 +
-CAUTION: Do not set an object storage property to an empty string `""` or to `null` as a way to reset it to its default value.
+include::manage:partial$kubernetes/reset-object-storage-properties.adoc[]
+
+include::manage:partial$kubernetes/extraclusterconfig.adoc[]
 
 include::manage:partial$tiered-storage/aws-kms-key.adoc[leveloffset=+4]
 
@@ -337,8 +345,10 @@ Replace the following placeholders:
 +
 - `<region>`: The region of your bucket.
 - `<redpanda-bucket-name>`: The name of your bucket.
++
+include::manage:partial$kubernetes/reset-object-storage-properties.adoc[]
 
-CAUTION: Do not set an object storage property to an empty string `""` or to `null` as a way to reset it to its default value.
+include::manage:partial$kubernetes/extraclusterconfig.adoc[]
 
 ===== **Configure access with access keys**
 
@@ -457,7 +467,9 @@ Replace the following placeholders:
 - `<region>`: The region of your bucket.
 - `<redpanda-bucket-name>`: The name of your bucket.
 +
-CAUTION: Do not set an object storage property to an empty string `""` or to `null` as a way to reset it to its default value.
+include::manage:partial$kubernetes/reset-object-storage-properties.adoc[]
+
+include::manage:partial$kubernetes/extraclusterconfig.adoc[]
 
 include::manage:partial$tiered-storage/gcp-kms-key.adoc[leveloffset=+4]
 
@@ -562,10 +574,12 @@ Replace the following placeholders:
 - `<container-name>`: The name of the Azure container in your Azure account.
 - `<managed-identity-client-id>`: The client ID for your Azure managed identity.
 --
++
+include::manage:partial$kubernetes/reset-object-storage-properties.adoc[]
 
-NOTE: The `serviceAccount` annotations and the `statefulset` Pod labels are essential for the Azure webhook to inject the necessary Azure-specific environment variables and the projected service account token volume into the pods. For more information, visit https://learn.microsoft.com/en-us/azure/aks/workload-identity-overview?tabs=dotnet[Microsoft Entra Workload ID with Azure Kubernetes Service (AKS)^].
+The `serviceAccount` annotations and the `statefulset` Pod labels are essential for the Azure webhook to inject the necessary Azure-specific environment variables and the projected service account token volume into the pods. For more information, visit https://learn.microsoft.com/en-us/azure/aks/workload-identity-overview?tabs=dotnet[Microsoft Entra Workload ID with Azure Kubernetes Service (AKS)^].
 
-CAUTION: Do not set an object storage property to an empty string `""` or to `null` as a way to reset it to its default value.
+include::manage:partial$kubernetes/extraclusterconfig.adoc[]
 
 ===== **Configure access with account access keys**
 
@@ -666,7 +680,9 @@ Replace the following placeholders:
 - `<account-name>`: The name of your Azure account.
 - `<container-name>`: The name of the Azure container in your Azure account.
 +
-CAUTION: Do not set an object storage property to an empty string `""` or to `null` as a way to reset it to its default value.
+include::manage:partial$kubernetes/reset-object-storage-properties.adoc[]
+
+include::manage:partial$kubernetes/extraclusterconfig.adoc[]
 
 - For information about how to grant access from an internet IP range (if you need to open additional routes/ports between your broker nodes and Azure Blob Storage; for example, in a hybrid cloud deployment), see the https://learn.microsoft.com/en-us/azure/storage/common/storage-network-security?toc=%2Fazure%2Fstorage%2Fblobs%2Ftoc.json&bc=%2Fazure%2Fstorage%2Fblobs%2Fbreadcrumb%2Ftoc.json&tabs=azure-portal#grant-access-from-an-internet-ip-range[Microsoft documentation^].
 

modules/manage/partials/whole-cluster-restore.adoc

@@ -152,6 +152,8 @@ helm upgrade --install redpanda redpanda/redpanda --namespace <namespace> --crea
 
 - `storage.tiered`: Make sure to configure the target cluster with the same Tiered Storage settings as the failed source cluster.
 - `config.cluster.cloud_storage_attempt_cluster_restore_on_bootstrap`: Automate cluster restore in Kubernetes. Setting to `true` is recommended when using an automated method for deployment. When bootstrapping a cluster with a given bucket, make sure that any previous cluster using the bucket is fully destroyed, otherwise Tiered Storage subsystems may interfere with each other.
+
+include::manage:partial$kubernetes/extraclusterconfig.adoc[]
 endif::[]
 
 ifndef::env-kubernetes[]