BYOVPC support for secrets and Iceberg REST catalogs #313
Conversation
✅ Deploy Preview for rp-cloud ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
|
Important Review skippedAuto incremental reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the You can disable this status message by setting the 📝 WalkthroughWalkthroughThis pull request updates and extends the documentation related to enabling and managing secrets management for Bring Your Own VPC (BYOVPC) clusters on both AWS and GCP in Redpanda Cloud. It adds a new guide for enabling secrets management on existing BYOVPC clusters on GCP, clarifies IAM role and service account setup for both AWS and GCP, and updates navigation to include the new documentation. Additional notes clarify the status of the Iceberg integration and how to reference secrets in cluster configuration. Sequence Diagram(s)sequenceDiagram
participant User
participant GCP Console/CLI
participant Redpanda Cloud API
participant Redpanda Cloud UI
User->>GCP Console/CLI: Create redpanda-operator service account
User->>GCP Console/CLI: Assign custom IAM roles to service accounts
User->>GCP Console/CLI: Bind workload identity user role
User->>Redpanda Cloud API: PATCH /v1/clusters/{cluster-id} to add operator service account
User->>Redpanda Cloud UI: Verify secrets management enabled
Assessment against linked issues
Possibly related PRs
Suggested reviewers
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
modules/get-started/pages/cluster-types/byoc/gcp/enable-secrets-byovpc-gcp.adoc
Outdated
Show resolved
Hide resolved
modules/get-started/pages/cluster-types/byoc/gcp/enable-secrets-byovpc-gcp.adoc
Outdated
Show resolved
Hide resolved
modules/get-started/pages/cluster-types/byoc/gcp/enable-secrets-byovpc-gcp.adoc
Show resolved
Hide resolved
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🔭 Outside diff range comments (2)
modules/get-started/pages/cluster-types/byoc/gcp/vpc-byo-gcp.adoc (2)
32-38: 🛠️ Refactor suggestionFix AsciiDoc literal block delimiters
The subnet list is wrapped with--delimiters, but AsciiDoc requires four-character fences (----) for listing or literal blocks. This could render incorrectly. Either remove the delimiters to allow normal bullet formatting or replace--/--with----/----.
47-47:⚠️ Potential issueTypo in placeholder name
The placeholder<secondary-ipv2-range-name-for-services>appears to be a typo. It should read<secondary-ipv4-range-name-for-services>to match the IPv4 context.
🧹 Nitpick comments (7)
modules/manage/pages/cluster-maintenance/config-cluster.adoc (2)
38-43: Encapsulate CLI secret notation in an admon block
Great addition demonstrating secrets in therpkCLI. For consistency and better visibility, wrap this instruction and code in a proper Asciidoc admon block:[NOTE] ==== To set a cluster property with a secret, run: [source,bash] ---- rpk cluster config set iceberg_rest_catalog_client_secret ${secrets.<secret-name>} ---- ====
82-96: Encapsulate API secret notation in an admon block
The Cloud API example is clear; consider formatting it similarly within an admon block for consistency:[NOTE] ==== To set a cluster property with a secret via API: [source,bash] ---- curl -H "Authorization: Bearer <token>" -X PATCH \ "https://api.cloud.redpanda.com/v1/clusters/<cluster-id>" \ -H 'accept: application/json' \ -H 'content-type: application/json' \ -d '{"cluster_configuration":{"custom_properties":{"iceberg_rest_catalog_client_secret":"${secrets.<secret-name>}"}}}' ---- ====modules/manage/pages/iceberg/about-iceberg-topics.adoc (1)
5-5: Format beta note as an admon block
The beta notice is important but should leverage Asciidoc’s admon syntax. For example:[NOTE] ==== The Iceberg integration for Redpanda Cloud is a beta feature. It is not supported for production deployments. ====modules/get-started/pages/cluster-types/byoc/aws/vpc-byo-aws.adoc (1)
68-68: Minor formatting cleanup
Removing the trailing space after the Redpanda Connect bullet is a good formatting fix.modules/get-started/pages/cluster-types/byoc/gcp/enable-secrets-byovpc-gcp.adoc (3)
1-4: Validate:page-beta:attribute usage
The:page-beta:attribute is declared without a value. Confirm if this is intended (e.g., some pages annotate with a note) or remove it if it has no effect in your Antora build.
14-18: Attach collapsible block correctly under list item
To nest the collapsible block beneath the step, place the+immediately after the list item line and ensure subsequent block markers (====) start on the next line—without preceding list dots. Also, change.Show commandsto. Show commands(add a space) to conform to AsciiDoc list syntax.
19-23: Remove redundant comment line in snippet
The first line inside the code block is a lone#. Please remove it to avoid confusion and keep the snippet focused on the essential commands.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (7)
local-antora-playbook.yml(1 hunks)modules/ROOT/nav.adoc(1 hunks)modules/get-started/pages/cluster-types/byoc/aws/vpc-byo-aws.adoc(2 hunks)modules/get-started/pages/cluster-types/byoc/gcp/enable-secrets-byovpc-gcp.adoc(1 hunks)modules/get-started/pages/cluster-types/byoc/gcp/vpc-byo-gcp.adoc(2 hunks)modules/manage/pages/cluster-maintenance/config-cluster.adoc(2 hunks)modules/manage/pages/iceberg/about-iceberg-topics.adoc(1 hunks)
🔇 Additional comments (5)
modules/ROOT/nav.adoc (1)
22-22: Navigation link insertion is correct
The new BYOVPC GCP secrets management link is added at the appropriate nesting level and aligns with existing structure.modules/get-started/pages/cluster-types/byoc/aws/vpc-byo-aws.adoc (2)
21-26: Secrets management note block is well-formed
The[NOTE]block clearly communicates default secrets support and remediation steps. It follows Asciidoc admon conventions.
70-75: JSON snippet insertion is clear
The example forcondition_tagscorrectly uses the+join and a code block, enhancing the Terraform customization guidance.modules/get-started/pages/cluster-types/byoc/gcp/enable-secrets-byovpc-gcp.adoc (2)
81-84: Verify workload-identity binding syntax
Thegcloud iam service-accounts add-iam-policy-bindingcommand for the operator SA uses placeholders for namespace and account ID. Double-check that the member string (serviceAccount:<service-project-id>.svc.id.goog[redpanda-system/<redpanda_operator-gcp-sa-account-id>]) matches your cluster’s namespace and SA naming conventions.
87-108: Check xref anchor for PATCH endpoint
The cross-referencexref:api:ROOT:cloud-controlplane-api.adoc#patch-/v1/clusters/-cluster.id-may not match the actual anchor in your API docs. Please verify that the link resolves correctly in the rendered documentation.
local-antora-playbook.yml
Outdated
| branches: HEAD | ||
| - url: https://github.com/redpanda-data/documentation | ||
| branches: [main, v/*, api, shared, site-search] | ||
| branches: ['DOC-1222-iceberg-byovpc-updates', v/*, api, shared, site-search] |
There was a problem hiding this comment.
Ensure wildcard branch patterns are correctly quoted
The branches list includes v/* unquoted, which may lead to YAML parsing errors. Wrap wildcard patterns in quotes (e.g. 'v/*').
🤖 Prompt for AI Agents
In local-antora-playbook.yml at line 18, the wildcard branch pattern v/* is not
quoted, which can cause YAML parsing errors. Fix this by enclosing the wildcard
pattern in single quotes, changing v/* to 'v/*' within the branches list.
| :description: Learn how Redpanda can integrate topics with Apache Iceberg. | ||
| :page-beta: true | ||
|
|
||
| NOTE: The Iceberg integration for Redpanda Cloud is a beta feature. It is not supported for production deployments. |
There was a problem hiding this comment.
| NOTE: The Iceberg integration for Redpanda Cloud is a beta feature. It is not supported for production deployments. | |
| NOTE: The Iceberg integration for Redpanda Cloud is a beta feature. It is not supported for production deployments. |
@simon0191 This note used to be on https://docs.redpanda.com/redpanda-cloud/manage/iceberg/use-iceberg-catalogs/ because of previous lack of support for secrets. Do we still want to add a note about using Iceberg in production while it is in beta for Cloud?
There was a problem hiding this comment.
I'd replace this with the catalog support matrix mentioning that each integration has different maturity level
There was a problem hiding this comment.
@simon0191 Let's address this in a separate PR for https://redpandadata.atlassian.net/browse/DOC-1411 so PM and ideally DevProd can vet it as well.
modules/get-started/pages/cluster-types/byoc/gcp/enable-secrets-byovpc-gcp.adoc
Show resolved
Hide resolved
|
|
||
| [IMPORTANT] | ||
| ==== | ||
| BYOVPC is an add-on feature that may require an additional purchase. To unlock this feature for your account, contact your Redpanda account team or https://www.redpanda.com/price-estimator[Redpanda Sales^]. |
There was a problem hiding this comment.
So it may or may not require purchase? Or does it absolutely require purchase to unlock? The word "may" here makes me think, maybe/maybe not, which doesn't give me clarity on this.
There was a problem hiding this comment.
@david-yu would you be able to clarify or give guidance on the language for BYOVPC access?
kbatuigas
force-pushed
the
DOC-1222-Document-feature-Iceberg-on-Cloud-BYOVPC-GA-AWS-GCP
branch
from
a1fdcab to
8bb1f1e
Compare
| BYOVPC is an add-on feature that may require an additional purchase. To unlock this feature for your account, contact your Redpanda account team or https://www.redpanda.com/price-estimator[Redpanda Sales^]. | ||
| ==== | ||
|
|
||
| Storing secrets in your cluster allows you to keep your cloud infrastructure secure as you integrate your data across different systems, for example, REST catalogs with your Iceberg-enabled topics. To enable secrets management on an existing BYOVPC cluster, you must update your cluster configuration. You can also create xref:get-started:cluster-types/byoc/gcp/vpc-byo-gcp.adoc[a new BYOVPC cluster] with secrets management already enabled. |
There was a problem hiding this comment.
"if you haven't done it already"
Basically, this applies only to existing clusters, but future clusters will have secrets enabled
| :description: Learn how Redpanda can integrate topics with Apache Iceberg. | ||
| :page-beta: true | ||
|
|
||
| NOTE: The Iceberg integration for Redpanda Cloud is a beta feature. It is not supported for production deployments. |
There was a problem hiding this comment.
I'd replace this with the catalog support matrix mentioning that each integration has different maturity level
| == Update cluster configuration | ||
|
|
||
| To update your cluster configuration properties, make a request to the xref:api:ROOT:cloud-controlplane-api.adoc#patch-/v1/clusters/-id-[`PATCH /v1/clusters/\{id}`] endpoint, passing the cluster ID as a parameter. Include the properties to update in the request body. | ||
| To update your cluster configuration properties, make a request to the xref:api:ROOT:cloud-controlplane-api.adoc#patch-/v1/clusters/-cluster.id-[`PATCH /v1/clusters/\{id}`] endpoint, passing the cluster ID as a parameter. Include the properties to update in the request body. |
There was a problem hiding this comment.
In here let's also show how to do it with rpk which is much better UX
There was a problem hiding this comment.
@simon0191 This is the page that guides readers on how to do specific tasks with the Control Plane API - we should add the rpk instructions on the general Iceberg doc https://docs.redpanda.com/redpanda-cloud/manage/iceberg/about-iceberg-topics/#enable-iceberg-integration (per https://redpandadata.atlassian.net/browse/DOC-1413)
| [source,bash] | ||
| ---- | ||
| curl -H "Authorization: Bearer <token>" -X PATCH \ |
There was a problem hiding this comment.
Let's also show how to do it with rpk which is much better UX
There was a problem hiding this comment.
@simon0191 This is the general doc for configuring cluster properties - the rpk instructions are described in the same doc https://github.com/redpanda-data/cloud-docs/pull/313/files#diff-c8fbb49e2369d966902a69910a95b4e432a6931f6efaf898897f416acbd9cd8dR26-R45 (preview here). The example to enable Iceberg should be added in the general Iceberg doc , per https://redpandadata.atlassian.net/browse/DOC-1413. I'll do so in a separate PR.
…ote to About doc instead
| ==== | ||
| Secrets management is enabled by default with the Terraform code in the example repository. It allows you to store and read secrets in your cluster, for example to integrate a REST catalog with Iceberg-enabled topics. | ||
| For existing BYOVPC clusters, contact https://support.redpanda.com/hc/en-us/requests/new[Redpanda Support^] to enable secrets management. |
There was a problem hiding this comment.
@wzzzrd86 just a heads up that per @tomasz-sadura , existing AWS BYOVPC clusters will need assistance from Support to enable secrets management.
kbatuigas
force-pushed
the
DOC-1222-Document-feature-Iceberg-on-Cloud-BYOVPC-GA-AWS-GCP
branch
from
2b8b12f to
110e14a
Compare
kbatuigas
changed the title
kbatuigas
deleted the
DOC-1222-Document-feature-Iceberg-on-Cloud-BYOVPC-GA-AWS-GCP
branch
Description
Related PR: redpanda-data/docs#1140
This pull request introduces support for secrets management in BYOVPC clusters on AWS and GCP, along with related documentation updates and improvements. It includes instructions for enabling secrets management on new and existing clusters, as well as examples for configuring cluster properties with secrets. Below are the most important changes grouped by theme:
Secrets Management for BYOVPC Clusters:
enable-secrets-byovpc-gcp.adocwith detailed steps to enable secrets management on existing BYOVPC clusters on GCP, including creating service accounts, assigning roles, and updating cluster configurations.vpc-byo-gcp.adocto include commands for creating and binding roles for Redpanda operator and cluster service accounts to support secrets management. [1] [2]vpc-byo-aws.adocindicating that secrets management is enabled by default in the Terraform example code for AWS and providing instructions for enabling it on existing clusters.Documentation Updates:
nav.adocto include a link to the new secrets management page for GCP.whats-new-cloud.adocannouncing secrets management support for BYOVPC clusters on AWS and GCP.config-cluster.adocto include examples of setting cluster properties with secrets usingrpkand API commands. [1] [2]Miscellaneous Improvements:
about-iceberg-topics.adoc.controlplane-api.adoc.These changes provide enhanced functionality for managing secrets securely in BYOVPC clusters and improve the clarity and usability of the documentation.
Resolves https://github.com/redpanda-data/documentation-private/issues/
Review deadline: 30 May
Page previews
What's New in Cloud
Create a BYOVPC Cluster on GCP
Enable Secrete Management on an Existing BYOVPC Cluster on GCP
Create a BYOVPC Cluster on AWS
Configure Cluster Properties > Set cluster configuration values
Checks