Skip to content
Merged
26 changes: 26 additions & 0 deletions modules/manage/partials/iceberg/about-iceberg-topics.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -150,6 +150,9 @@ SUBJECT VERSION ID TYPE
----

ifdef::env-cloud[]

=== Access Iceberg data

To query the Iceberg table, you need access to the object storage bucket or container where the Iceberg data is stored. For BYOC clusters, the bucket name and table location are as follows:

|===
Expand All @@ -170,6 +173,29 @@ The Redpanda cluster ID is also used as the container name (ID) and the storage


|===

For Azure clusters, you must add the public IP addresses or ranges from the REST catalog service, or other clients requiring access to the Iceberg data, to your cluster's allow list. Alternatively, add subnet IDs to the allow list if the requests originate from the same Azure region.

For example, to add subnet IDs to the allow list through the Control Plane API xref:api:ROOT:cloud-controlplane-api.adoc#patch-/v1/clusters/-cluster.id-[`PATCH /v1/clusters/<cluster-id>`] endpoint, run:

[,bash]
----
curl -X PATCH https://api.cloud.redpanda.com/v1/clusters/<cluster-id> \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${RP_CLOUD_TOKEN}" \
-d @- << EOF
{
"cloud_storage": {
"azure": {
"allowed_subnet_ids": [
<list-of-subnet-ids>
]
}
}
}
EOF
----

endif::[]

As you produce records to the topic, the data also becomes available in object storage for Iceberg-compatible clients to consume. You can use the same analytical tools to xref:manage:iceberg/query-iceberg-topics.adoc[read the Iceberg topic data] in a data lake as you would for a relational database.
Expand Down
23 changes: 23 additions & 0 deletions modules/manage/partials/iceberg/query-iceberg-topics.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,29 @@ The Redpanda cluster ID is also used as the container name (ID) and the storage
| `redpanda-cloud-storage-<cluster-id>`

|===

For Azure clusters, you must add the public IP addresses or ranges from the REST catalog service, or other clients requiring access to the Iceberg data, to your cluster's allow list. Alternatively, add subnet IDs to the allow list if the requests originate from the same Azure region.

For example, to add subnet IDs to the allow list through the Control Plane API xref:api:ROOT:cloud-controlplane-api.adoc#patch-/v1/clusters/-cluster.id-[`PATCH /v1/clusters/<cluster-id>`] endpoint, run:

[,bash]
----
curl -X PATCH https://api.cloud.redpanda.com/v1/clusters/<cluster-id> \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${RP_CLOUD_TOKEN}" \
-d @- << EOF
{
"cloud_storage": {
"azure": {
"allowed_subnet_ids": [
<list-of-subnet-ids>
]
}
}
}
EOF
----

endif::[]

If your engine needs the full JSON metadata path, use the following:
Expand Down
17 changes: 13 additions & 4 deletions modules/manage/partials/iceberg/use-iceberg-catalogs.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -3,10 +3,6 @@ ifndef::env-cloud[:about-iceberg-doc: manage:iceberg/topic-iceberg-integration.a

To read from the Redpanda-generated xref:{about-iceberg-doc}[Iceberg table], your Iceberg-compatible client or tool needs access to the catalog to retrieve the table metadata and know the current state of the table. The catalog provides the current table metadata, which includes locations for all the table's data files. You can configure Redpanda to either connect to a REST-based catalog, or use a filesystem-based catalog.

ifdef::env-cloud[]
NOTE: The Iceberg integration for Redpanda Cloud is a beta feature. It is not supported for production deployments. To configure REST catalog authentication for use with Iceberg topics in your cloud cluster, contact https://support.redpanda.com/hc/en-us/requests/new[Redpanda support^].
endif::[]

For production deployments, Redpanda recommends using an external REST catalog to manage Iceberg metadata. This enables built-in table maintenance, safely handles multiple engines and tools accessing tables at the same time, facilitates data governance, and maximizes data discovery. However, if it is not possible to use a REST catalog, you may use the filesystem-based catalog (`object_storage` catalog type), which does not require you to maintain a separate service to access the Iceberg data. In either case, you use the catalog to load, query, or refresh the Iceberg table as you produce to the Redpanda topic. See the documentation for your query engine or Iceberg-compatible tool for specific guidance on adding the Iceberg tables to your data warehouse or lakehouse using the catalog.

After you have selected a catalog type at the cluster level and xref:{about-iceberg-doc}#enable-iceberg-integration[enabled the Iceberg integration] for a topic, you cannot switch to another catalog type.
Expand All @@ -15,6 +11,19 @@ After you have selected a catalog type at the cluster level and xref:{about-iceb

Connect to an Iceberg REST catalog using the standard https://github.com/apache/iceberg/blob/main/open-api/rest-catalog-open-api.yaml[REST API^] supported by many catalog providers. Use this catalog integration type with REST-enabled Iceberg catalog services, such as https://docs.databricks.com/en/data-governance/unity-catalog/index.html[Databricks Unity^] and https://other-docs.snowflake.com/en/opencatalog/overview[Snowflake Open Catalog^].

ifdef::env-cloud[]
=== Prerequisites

For BYOVPC clusters, you must:

. Enable secrets management, which allows you to store and use secrets in your cluster's Iceberg catalog authentication properties.
+
Secrets management is enabled by default for AWS if you follow the guide to xref:get-started:cluster-types/byoc/aws/vpc-byo-aws.adoc[creating a new BYOVPC cluster]. For GCP, follow the guides to enable secrets management for a xref:get-started:cluster-types/byoc/gcp/vpc-byo-gcp.adoc[new BYOVPC cluster] or an xref:get-started:cluster-types/byoc/gcp/enable-secrets-byovpc-gcp.adoc[existing BYOVPC cluster].
. Ensure that your network security settings allow egress traffic from the Redpanda network to the catalog service endpoints.
endif::[]

=== Set cluster properties

To connect to a REST catalog, set the following cluster configuration properties:

* config_ref:iceberg_catalog_type,true,properties/cluster-properties[`iceberg_catalog_type`]: `rest`
Expand Down