From 742e63085af9e2acb508ccacc265375e9e11c6b8 Mon Sep 17 00:00:00 2001 From: kbatuigas <36839689+kbatuigas@users.noreply.github.com> Date: Tue, 20 May 2025 16:16:08 -0700 Subject: [PATCH 1/7] Single source Snowflake doc with cloud-only content --- ...anda-topics-iceberg-snowflake-catalog.adoc | 42 +++++++++++++++++++ 1 file changed, 42 insertions(+) diff --git a/modules/manage/pages/iceberg/redpanda-topics-iceberg-snowflake-catalog.adoc b/modules/manage/pages/iceberg/redpanda-topics-iceberg-snowflake-catalog.adoc index 2160d63d8..9118c1768 100644 --- a/modules/manage/pages/iceberg/redpanda-topics-iceberg-snowflake-catalog.adoc +++ b/modules/manage/pages/iceberg/redpanda-topics-iceberg-snowflake-catalog.adoc @@ -7,13 +7,20 @@ include::shared:partial$enterprise-license.adoc[] ==== +// tag::single-source[] + This guide walks you through querying Redpanda topics as Iceberg tables in https://docs.snowflake.com/en/user-guide/tables-iceberg[Snowflake^], with AWS S3 as object storage and a catalog integration using https://other-docs.snowflake.com/en/opencatalog/overview[Open Catalog^]. == Prerequisites +ifdef::env-cloud[] +* `rpk` or familiarity with the Redpanda Cloud API to use secrets in your cluster configuration. For `rpk`, see xref:manage:rpk/rpk-install.adoc[]. For the Cloud API, you must xref:manage:api/cloud-api-authentication.adoc[authenticate] using a service account. +endif::[] +ifndef::env-cloud[] * xref:manage:tiered-storage.adoc#configure-object-storage[Object storage configured] for your cluster and xref:manage:tiered-storage.adoc#enable-tiered-storage[Tiered Storage enabled] for the topics for which you want to generate Iceberg tables. + You need the S3 bucket URI, so you can configure it as external storage for Open Catalog. +endif::[] * A Snowflake account. * An Open Catalog account. To https://other-docs.snowflake.com/en/opencatalog/create-open-catalog-account[create an Open Catalog account^], you require ORGADMIN access in Snowflake. * An internal catalog created in Open Catalog with your Tiered Storage AWS S3 bucket configured as external storage. @@ -58,6 +65,38 @@ Grant privileges to the principal created in the previous step: To configure your Redpanda cluster to enable Iceberg on a topic and integrate with Open Catalog: +ifdef::env-cloud[] +. xref:manage:iceberg/use-iceberg-catalogs.adoc#store-a-secret-for-rest-catalog-authentication[Store the Open Catalog client secret in your cluster] using `rpk` or the Data Plane API. +. Edit your cluster configuration to set the `iceberg_enabled` property to `true`, and set the catalog integration properties listed in the example below using `rpk` or the Control Plane API. For example, to use `rpk cluster config set`, run the following: ++ +[,bash] +---- +rpk cluster config set iceberg_enabled=true iceberg_catalog_type=rest +iceberg_rest_catalog_endpoint=https://-.snowflakecomputing.com/polaris/api/catalog iceberg_rest_catalog_authentication_mode=oauth2 iceberg_rest_catalog_client_id= iceberg_rest_catalog_client_secret=${secrets.} +iceberg_rest_catalog_prefix: + +# Optional properties: +# iceberg_translation_interval_ms_default=1000 +# iceberg_catalog_commit_interval_ms=1000 +---- ++ +Use your own values for the following placeholders: ++ +-- +- `` and ``: Your https://docs.snowflake.com/en/sql-reference/sql/create-catalog-integration-open-catalog#required-parameters[Open Catalog account URI^] is composed of these values. ++ +TIP: In Snowflake, navigate to **Admin**, then **Accounts**. Click the ellipsis near your Open Catalog account name, and select **Manage URLs**. The **Current URL** contains `` and ``. +- ``: The client ID of the service connection you created in an earlier step. +- ``: The name of the secret you created in the previous step. You must pass the secret name to the `${secrets.}` placeholder, not the secret value. +- ``: The name of your catalog in Open Catalog. +-- ++ +[,bash,role=no-copy] +---- +Successfully updated configuration. New configuration version is 2. +---- +endif::[] +ifndef::env-cloud[] . Edit your cluster configuration to set the `iceberg_enabled` property to `true`, and set the catalog integration properties listed in the example below. You must restart your cluster if you change this configuration for a running cluster. You can run `rpk cluster config edit` to update these properties: + [,bash] @@ -92,6 +131,7 @@ Successfully updated configuration. New configuration version is 2. ---- . You must restart your cluster so that the configuration changes take effect. +endif::[] . Enable the integration for a topic by configuring the topic property `redpanda.iceberg.mode`. This mode creates an Iceberg table for the topic consisting of two columns, one for the record metadata including the key, and another binary column for the record's value. See xref:manage:iceberg/about-iceberg-topics.adoc#enable-iceberg-integration[Enable Iceberg integration] for more details on Iceberg modes. The following examples show how to use xref:get-started:rpk-install.adoc[`rpk`] to either create a new topic, or alter the configuration for an existing topic, to set the Iceberg mode to `key_value`. + @@ -218,3 +258,5 @@ Your query results should look like the following: +--------------------------------------------------------------------------------------------------------------+------------+ ---- + +// end::single-source[] \ No newline at end of file From a0e96a47524cb3140b2f635b04adc04019ea1c8d Mon Sep 17 00:00:00 2001 From: kbatuigas <36839689+kbatuigas@users.noreply.github.com> Date: Tue, 20 May 2025 16:55:12 -0700 Subject: [PATCH 2/7] How to configure topic property in Cloud --- ...edpanda-topics-iceberg-snowflake-catalog.adoc | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-) diff --git a/modules/manage/pages/iceberg/redpanda-topics-iceberg-snowflake-catalog.adoc b/modules/manage/pages/iceberg/redpanda-topics-iceberg-snowflake-catalog.adoc index 9118c1768..8df9c56ce 100644 --- a/modules/manage/pages/iceberg/redpanda-topics-iceberg-snowflake-catalog.adoc +++ b/modules/manage/pages/iceberg/redpanda-topics-iceberg-snowflake-catalog.adoc @@ -87,7 +87,7 @@ Use your own values for the following placeholders: + TIP: In Snowflake, navigate to **Admin**, then **Accounts**. Click the ellipsis near your Open Catalog account name, and select **Manage URLs**. The **Current URL** contains `` and ``. - ``: The client ID of the service connection you created in an earlier step. -- ``: The name of the secret you created in the previous step. You must pass the secret name to the `${secrets.}` placeholder, not the secret value. +- ``: The name of the secret you created in the previous step. You must pass the secret name to the `${secrets.}` placeholder, not the secret value itself. - ``: The name of your catalog in Open Catalog. -- + @@ -133,7 +133,19 @@ Successfully updated configuration. New configuration version is 2. . You must restart your cluster so that the configuration changes take effect. endif::[] -. Enable the integration for a topic by configuring the topic property `redpanda.iceberg.mode`. This mode creates an Iceberg table for the topic consisting of two columns, one for the record metadata including the key, and another binary column for the record's value. See xref:manage:iceberg/about-iceberg-topics.adoc#enable-iceberg-integration[Enable Iceberg integration] for more details on Iceberg modes. The following examples show how to use xref:get-started:rpk-install.adoc[`rpk`] to either create a new topic, or alter the configuration for an existing topic, to set the Iceberg mode to `key_value`. +. Enable the integration for a topic by configuring the topic property `redpanda.iceberg.mode`. This mode creates an Iceberg table for the topic consisting of two columns, one for the record metadata including the key, and another binary column for the record's value. See xref:manage:iceberg/about-iceberg-topics.adoc#enable-iceberg-integration[Enable Iceberg integration] for more details on Iceberg modes. +ifdef::env-cloud[] ++ +Use any of the following to set `redpanda.iceberg.mode`: ++ +-- +* `rpk`. See the examples below for running the `rpk topic` command. +* The Cloud UI. Navigate to *Topics* to create a new topic and specify `redpanda.iceberg.mode` in *Additional Configuration*, or edit an existing topic under the topic's *Configuration* tab. +* The Data Plane API to xref:api:ROOT:cloud-dataplane-api.adoc#post-/v1/topics[create a new topic] or xref:api:ROOT:cloud-dataplane-api.adoc#patch-/v1/topics/-topic_name-/configurations[update a property for an existing topic]. Specify the key-value pair for `redpanda.iceberg.mode` in the request body. +-- +endif::[] ++ +The following examples show how to use xref:get-started:rpk-install.adoc[`rpk`] to either create a new topic, or alter the configuration for an existing topic, to set the Iceberg mode to `key_value`. + .Create a new topic and set `redpanda.iceberg.mode`: [,bash] From feb9b244ca2f7e68da7562b000c42003c6a90b7d Mon Sep 17 00:00:00 2001 From: kbatuigas <36839689+kbatuigas@users.noreply.github.com> Date: Tue, 20 May 2025 17:00:55 -0700 Subject: [PATCH 3/7] Minor edit --- .../iceberg/redpanda-topics-iceberg-snowflake-catalog.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/manage/pages/iceberg/redpanda-topics-iceberg-snowflake-catalog.adoc b/modules/manage/pages/iceberg/redpanda-topics-iceberg-snowflake-catalog.adoc index 8df9c56ce..7e93061f7 100644 --- a/modules/manage/pages/iceberg/redpanda-topics-iceberg-snowflake-catalog.adoc +++ b/modules/manage/pages/iceberg/redpanda-topics-iceberg-snowflake-catalog.adoc @@ -139,7 +139,7 @@ ifdef::env-cloud[] Use any of the following to set `redpanda.iceberg.mode`: + -- -* `rpk`. See the examples below for running the `rpk topic` command. +* `rpk`. See the examples below for running the `rpk topic` commands. * The Cloud UI. Navigate to *Topics* to create a new topic and specify `redpanda.iceberg.mode` in *Additional Configuration*, or edit an existing topic under the topic's *Configuration* tab. * The Data Plane API to xref:api:ROOT:cloud-dataplane-api.adoc#post-/v1/topics[create a new topic] or xref:api:ROOT:cloud-dataplane-api.adoc#patch-/v1/topics/-topic_name-/configurations[update a property for an existing topic]. Specify the key-value pair for `redpanda.iceberg.mode` in the request body. -- From 646a70fc0d45e37586fc6c043d0930ed408f78e2 Mon Sep 17 00:00:00 2001 From: kbatuigas <36839689+kbatuigas@users.noreply.github.com> Date: Tue, 20 May 2025 17:03:52 -0700 Subject: [PATCH 4/7] Deploy preview for Cloud page --- local-antora-playbook.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/local-antora-playbook.yml b/local-antora-playbook.yml index 0f53a55d6..3a1538c88 100644 --- a/local-antora-playbook.yml +++ b/local-antora-playbook.yml @@ -17,7 +17,7 @@ content: - url: https://github.com/redpanda-data/docs branches: [v/*, api, shared, site-search,'!v-end-of-life/*'] - url: https://github.com/redpanda-data/cloud-docs - branches: 'main' + branches: 'DOC-1389-cloud-snowflake-secrets-doc' - url: https://github.com/redpanda-data/redpanda-labs branches: main start_paths: [docs,'*/docs'] From d6e60693e3cccff178e8eea2883405c833cb345f Mon Sep 17 00:00:00 2001 From: kbatuigas <36839689+kbatuigas@users.noreply.github.com> Date: Tue, 20 May 2025 17:25:15 -0700 Subject: [PATCH 5/7] Apply suggestions from automated review --- .../redpanda-topics-iceberg-snowflake-catalog.adoc | 9 ++------- 1 file changed, 2 insertions(+), 7 deletions(-) diff --git a/modules/manage/pages/iceberg/redpanda-topics-iceberg-snowflake-catalog.adoc b/modules/manage/pages/iceberg/redpanda-topics-iceberg-snowflake-catalog.adoc index 7e93061f7..d87820491 100644 --- a/modules/manage/pages/iceberg/redpanda-topics-iceberg-snowflake-catalog.adoc +++ b/modules/manage/pages/iceberg/redpanda-topics-iceberg-snowflake-catalog.adoc @@ -19,7 +19,7 @@ endif::[] ifndef::env-cloud[] * xref:manage:tiered-storage.adoc#configure-object-storage[Object storage configured] for your cluster and xref:manage:tiered-storage.adoc#enable-tiered-storage[Tiered Storage enabled] for the topics for which you want to generate Iceberg tables. + -You need the S3 bucket URI, so you can configure it as external storage for Open Catalog. +You need the S3 bucket URI to configure it as external storage for Open Catalog. endif::[] * A Snowflake account. * An Open Catalog account. To https://other-docs.snowflake.com/en/opencatalog/create-open-catalog-account[create an Open Catalog account^], you require ORGADMIN access in Snowflake. @@ -27,11 +27,10 @@ endif::[] + Follow this guide to https://other-docs.snowflake.com/en/opencatalog/create-catalog#create-a-catalog-using-amazon-simple-storage-service-amazon-s3[create a catalog^] with the S3 bucket configured as external storage. You require admin permissions to carry out these steps in AWS: + --- . If you don't already have one, create an IAM policy that gives Open Catalog read and write access to your S3 bucket. . Create an IAM role and attach the IAM policy to the role. . After creating a new catalog in Open Catalog, grant the catalog's AWS IAM user access to the S3 bucket. --- ++ * A Snowflake https://docs.snowflake.com/en/user-guide/tables-iceberg-configure-external-volume[external volume^] set up using the Tiered Storage bucket. + Follow this guide to https://docs.snowflake.com/en/user-guide/tables-iceberg-configure-external-volume-s3[configure the external volume with S3^]. You can use the same IAM policy as the catalog for the external volume's IAM role and user. @@ -116,14 +115,12 @@ iceberg_catalog_commit_interval_ms: 1000 + Use your own values for the following placeholders: + --- - `` and ``: Your https://docs.snowflake.com/en/sql-reference/sql/create-catalog-integration-open-catalog#required-parameters[Open Catalog account URI^] is composed of these values. + TIP: In Snowflake, navigate to **Admin**, then **Accounts**. Click the ellipsis near your Open Catalog account name, and select **Manage URLs**. The **Current URL** contains `` and ``. - ``: The client ID of the service connection you created in an earlier step. - ``: The client secret of the service connection you created in an earlier step. - ``: The name of your catalog in Open Catalog. --- + [,bash,role=no-copy] ---- @@ -138,11 +135,9 @@ ifdef::env-cloud[] + Use any of the following to set `redpanda.iceberg.mode`: + --- * `rpk`. See the examples below for running the `rpk topic` commands. * The Cloud UI. Navigate to *Topics* to create a new topic and specify `redpanda.iceberg.mode` in *Additional Configuration*, or edit an existing topic under the topic's *Configuration* tab. * The Data Plane API to xref:api:ROOT:cloud-dataplane-api.adoc#post-/v1/topics[create a new topic] or xref:api:ROOT:cloud-dataplane-api.adoc#patch-/v1/topics/-topic_name-/configurations[update a property for an existing topic]. Specify the key-value pair for `redpanda.iceberg.mode` in the request body. --- endif::[] + The following examples show how to use xref:get-started:rpk-install.adoc[`rpk`] to either create a new topic, or alter the configuration for an existing topic, to set the Iceberg mode to `key_value`. From 157949fd8e95119beb49c20e1cf483d60f66e3b9 Mon Sep 17 00:00:00 2001 From: kbatuigas <36839689+kbatuigas@users.noreply.github.com> Date: Tue, 20 May 2025 17:27:02 -0700 Subject: [PATCH 6/7] Make rpk example more readable --- .../redpanda-topics-iceberg-snowflake-catalog.adoc | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/modules/manage/pages/iceberg/redpanda-topics-iceberg-snowflake-catalog.adoc b/modules/manage/pages/iceberg/redpanda-topics-iceberg-snowflake-catalog.adoc index d87820491..c52afcb77 100644 --- a/modules/manage/pages/iceberg/redpanda-topics-iceberg-snowflake-catalog.adoc +++ b/modules/manage/pages/iceberg/redpanda-topics-iceberg-snowflake-catalog.adoc @@ -70,9 +70,14 @@ ifdef::env-cloud[] + [,bash] ---- -rpk cluster config set iceberg_enabled=true iceberg_catalog_type=rest -iceberg_rest_catalog_endpoint=https://-.snowflakecomputing.com/polaris/api/catalog iceberg_rest_catalog_authentication_mode=oauth2 iceberg_rest_catalog_client_id= iceberg_rest_catalog_client_secret=${secrets.} -iceberg_rest_catalog_prefix: +rpk cluster config set \ + iceberg_enabled=true \ + iceberg_catalog_type=rest \ + iceberg_rest_catalog_endpoint=https://-.snowflakecomputing.com/polaris/api/catalog \ + iceberg_rest_catalog_authentication_mode=oauth2 \ + iceberg_rest_catalog_client_id= \ + iceberg_rest_catalog_client_secret=${secrets.} \ + iceberg_rest_catalog_prefix= # Optional properties: # iceberg_translation_interval_ms_default=1000 From 8ee8b29beef99432196b4abe76517bfc17ac8eb6 Mon Sep 17 00:00:00 2001 From: Kat Batuigas <36839689+kbatuigas@users.noreply.github.com> Date: Tue, 20 May 2025 22:30:58 -0700 Subject: [PATCH 7/7] Apply suggestions from code review Co-authored-by: Michele Cyran --- local-antora-playbook.yml | 2 +- .../redpanda-topics-iceberg-snowflake-catalog.adoc | 8 ++++---- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/local-antora-playbook.yml b/local-antora-playbook.yml index 3a1538c88..0f53a55d6 100644 --- a/local-antora-playbook.yml +++ b/local-antora-playbook.yml @@ -17,7 +17,7 @@ content: - url: https://github.com/redpanda-data/docs branches: [v/*, api, shared, site-search,'!v-end-of-life/*'] - url: https://github.com/redpanda-data/cloud-docs - branches: 'DOC-1389-cloud-snowflake-secrets-doc' + branches: 'main' - url: https://github.com/redpanda-data/redpanda-labs branches: main start_paths: [docs,'*/docs'] diff --git a/modules/manage/pages/iceberg/redpanda-topics-iceberg-snowflake-catalog.adoc b/modules/manage/pages/iceberg/redpanda-topics-iceberg-snowflake-catalog.adoc index c52afcb77..b51718ef8 100644 --- a/modules/manage/pages/iceberg/redpanda-topics-iceberg-snowflake-catalog.adoc +++ b/modules/manage/pages/iceberg/redpanda-topics-iceberg-snowflake-catalog.adoc @@ -66,7 +66,7 @@ To configure your Redpanda cluster to enable Iceberg on a topic and integrate wi ifdef::env-cloud[] . xref:manage:iceberg/use-iceberg-catalogs.adoc#store-a-secret-for-rest-catalog-authentication[Store the Open Catalog client secret in your cluster] using `rpk` or the Data Plane API. -. Edit your cluster configuration to set the `iceberg_enabled` property to `true`, and set the catalog integration properties listed in the example below using `rpk` or the Control Plane API. For example, to use `rpk cluster config set`, run the following: +. Edit your cluster configuration to set the `iceberg_enabled` property to `true`, and set the catalog integration properties listed in the example below using `rpk` or the Control Plane API. For example, to use `rpk cluster config set`, run: + [,bash] ---- @@ -135,17 +135,17 @@ Successfully updated configuration. New configuration version is 2. . You must restart your cluster so that the configuration changes take effect. endif::[] -. Enable the integration for a topic by configuring the topic property `redpanda.iceberg.mode`. This mode creates an Iceberg table for the topic consisting of two columns, one for the record metadata including the key, and another binary column for the record's value. See xref:manage:iceberg/about-iceberg-topics.adoc#enable-iceberg-integration[Enable Iceberg integration] for more details on Iceberg modes. +. Enable the integration for a topic by configuring the topic property `redpanda.iceberg.mode`. This mode creates an Iceberg table for the topic consisting of two columns: one for the record metadata including the key, and another binary column for the record's value. See xref:manage:iceberg/about-iceberg-topics.adoc#enable-iceberg-integration[Enable Iceberg integration] for more details on Iceberg modes. ifdef::env-cloud[] + Use any of the following to set `redpanda.iceberg.mode`: + -* `rpk`. See the examples below for running the `rpk topic` commands. +* `rpk`. See the following examples to run `rpk topic` commands. * The Cloud UI. Navigate to *Topics* to create a new topic and specify `redpanda.iceberg.mode` in *Additional Configuration*, or edit an existing topic under the topic's *Configuration* tab. * The Data Plane API to xref:api:ROOT:cloud-dataplane-api.adoc#post-/v1/topics[create a new topic] or xref:api:ROOT:cloud-dataplane-api.adoc#patch-/v1/topics/-topic_name-/configurations[update a property for an existing topic]. Specify the key-value pair for `redpanda.iceberg.mode` in the request body. endif::[] + -The following examples show how to use xref:get-started:rpk-install.adoc[`rpk`] to either create a new topic, or alter the configuration for an existing topic, to set the Iceberg mode to `key_value`. +The following examples show how to use `rpk` to create a new topic or alter the configuration for an existing topic, setting the Iceberg mode to `key_value`. + .Create a new topic and set `redpanda.iceberg.mode`: [,bash]