diff --git a/_data/menu_docs_stable.json b/_data/menu_docs_stable.json index 99c7d22cf3d..3a2e863d2ef 100644 --- a/_data/menu_docs_stable.json +++ b/_data/menu_docs_stable.json @@ -1082,6 +1082,10 @@ "page": "Overview", "url": "overview" }, + { + "page": "Iceberg REST Catalogs", + "url": "iceberg_rest_catalogs" + }, { "page": "Amazon S3 Tables", "url": "amazon_s3_tables" diff --git a/docs/stable/core_extensions/iceberg/amazon_s3_tables.md b/docs/stable/core_extensions/iceberg/amazon_s3_tables.md index 648f4cf1b12..5c81dea4112 100644 --- a/docs/stable/core_extensions/iceberg/amazon_s3_tables.md +++ b/docs/stable/core_extensions/iceberg/amazon_s3_tables.md @@ -4,21 +4,8 @@ redirect_from: null title: Amazon S3 Tables --- -The `iceberg` extension supports reading Iceberg tables stored in [Amazon S3 Tables](https://aws.amazon.com/s3/features/tables/). +The `iceberg` extension supports reading Iceberg tables stored in [Amazon S3 Tables](https://aws.amazon.com/s3/features/tables/). Make sure you have the Iceberg extension installed by following the steps at [Installing and Loading]({% link docs/stable/core_extensions/iceberg/overview.md %}) -## Requirements - -The S3 Tables support is currently experimental. -To use it, install the following extensions: - -```sql -FORCE INSTALL aws FROM core_nightly; -FORCE INSTALL httpfs FROM core_nightly; -FORCE INSTALL iceberg FROM core_nightly; -``` - -> If you want to switch back to using extensions from the `core` repository, -> follow the [extension documentation]({% link docs/stable/extensions/installing_extensions.md %}#force-installing-to-upgrade-extensions). ## Connecting to Amazon S3 Tables diff --git a/docs/stable/core_extensions/iceberg/amazon_sagemaker_lakehouse.md b/docs/stable/core_extensions/iceberg/amazon_sagemaker_lakehouse.md index fcb2d5cd67a..96eedf0305c 100644 --- a/docs/stable/core_extensions/iceberg/amazon_sagemaker_lakehouse.md +++ b/docs/stable/core_extensions/iceberg/amazon_sagemaker_lakehouse.md @@ -4,21 +4,8 @@ redirect_from: null title: Amazon SageMaker Lakehouse (AWS Glue) --- -The `iceberg` extension supports reading Iceberg tables through the [Amazon SageMaker Lakehouse (a.k.a. AWS Glue)](https://aws.amazon.com/sagemaker/lakehouse/) catalog. +The `iceberg` extension supports reading Iceberg tables through the [Amazon SageMaker Lakehouse (a.k.a. AWS Glue)](https://aws.amazon.com/sagemaker/lakehouse/) catalog. Make sure you have the Iceberg extension installed by following the steps at [Installing and Loading]({% link docs/stable/core_extensions/iceberg/overview.md %}) -## Requirements - -The S3 Tables support is currently experimental. -To use it, install the following extensions: - -```sql -FORCE INSTALL aws FROM core_nightly; -FORCE INSTALL httpfs FROM core_nightly; -FORCE INSTALL iceberg FROM core_nightly; -``` - -> If you want to switch back to using extensions from the `core` repository, -> follow the [extension documentation]({% link docs/stable/extensions/installing_extensions.md %}#force-installing-to-upgrade-extensions). ## Connecting to Amazon SageMaker Lakehouse diff --git a/docs/stable/core_extensions/iceberg/iceberg_rest_catalogs.md b/docs/stable/core_extensions/iceberg/iceberg_rest_catalogs.md new file mode 100644 index 00000000000..8eaf4193beb --- /dev/null +++ b/docs/stable/core_extensions/iceberg/iceberg_rest_catalogs.md @@ -0,0 +1,144 @@ +--- +layout: docu +redirect_from: null # maybe redirect from old amazon_s3_tables and amazon_sagemaker_lakehouse docs? +title: Iceberg REST Catalogs +--- + +The `iceberg` extension supports attaching Iceberg REST Catalogs. Before attaching an Iceberg REST Catalog, you must install the `iceberg` extension by following the instructions located in the [overview]({% link docs/stable/core_extensions/iceberg/overview.md %}). + +If you are attaching to an Iceberg REST Catalog managed by Amazon, please see the instructions for attaching to [Amazon S3 tables]({% link docs/stable/core_extensions/iceberg/amazon_s3_tables.md %}) or [Amazon Sagemaker Lakehouse]({% link docs/stable/core_extensions/iceberg/amazon_sagemaker_lakehouse.md %}). + +For all other Iceberg REST Catalogs, you can follow the instructions below. Please see the [Examples](#examples) section for questions about specific catalogs. + +Most Iceberg REST Catalogs authenticate via OAuth2. You can use the existing DuckDB secret workflow to store login credentials for the OAuth2 service. + +```sql +CREATE SECRET iceberg_secret ( + TYPE ICEBERG, + CLIENT_ID '⟨admin⟩', + CLIENT_SECRET '⟨password⟩', + OAUTH2_SERVER_URI '⟨http://irc_host_url.com/v1/oauth/tokens⟩' +); +``` + +If you already have a Bearer token, you can pass it directly to your `CREATE SECRET` statement + +```sql +CREATE SECRET iceberg_secret ( + TYPE ICEBERG, + TOKEN '⟨bearer_token⟩' +); +``` + +You can attach the Iceberg catalog with the following [`ATTACH`]({% link docs/stable/sql/statements/attach.md %}) statement. + +```sql +ATTACH '⟨warehouse⟩' AS iceberg_catalog ( + TYPE iceberg, + SECRET iceberg_secret, -- pass a specific secret name to prevent ambiguity + ENDPOINT ⟨https://rest_endpoint.com⟩ +); +``` + +To see the available tables run +```sql +SHOW ALL TABLES; +``` + +### ATTACH OPTIONS + +A REST Catalog with OAuth2 authorization can also be attached with just an `ATTACH` statement. See the complete list of `ATTACH` options for a REST catalog below. + + +| Parameter | Type | Default | Description | +| ---------------------------- | ---------- | -------- | ---------------------------------------------------------- | +| `ENDPOINT_TYPE` | `VARCHAR` | `NULL` | Used for attaching S3Tables or Glue catalogs. Allowed values are 'GLUE' and 'S3_TABLES' | +| `ENDPOINT` | `VARCHAR` | `NULL` | URL endpoint to communicate with the REST Catalog. Cannot be used in conjunction with `ENDPOINT_TYPE` | +| `SECRET` | `VARCHAR` | `NULL` | Name of secret used to communicate with the REST Catalog | +| `CLIENT_ID` | `VARCHAR` | `NULL` | CLIENT_ID used for Secret | +| `CLIENT_SECRET` | `VARCHAR` | `NULL` | CLIENT_SECRET needed for Secret | +| `DEFAULT_REGION` | `VARCHAR` | `NULL` | A Default region to use when communicating with the storage layer | +| `OAUTH2_SERVER_URI` | `VARCHAR` | `NULL` | OAuth2 server url for getting a Bearer Token | +| `AUTHORIZATION_TYPE` | `VARCHAR` | `OAUTH2` | Pass `SigV4` for Catalogs the require SigV4 authorization | + + +The following options can only be passed to a `CREATE SECRET` statement, and they require `AUTHORIZATION_TYPE` to be `OAUTH2` + +| Parameter | Type | Default | Description | +| ---------------------------- | ---------- | -------- | ---------------------------------------------------------- | +| `OAUTH2_GRANT_TYPE` | `VARCHAR` | `NULL` | Grant Type when requesting an OAuth Token | +| `OAUTH2_SCOPE` | `VARCHAR` | `NULL` | Requested scope for the returned OAuth Access Token | + +## Specific Catalog Examples + +### R2 Catalog + +To attach to an [R2 cloudflare](https://developers.cloudflare.com/r2/data-catalog/) managed catalog follow the attach steps below. + + +```sql +CREATE SECRET r2_secret ( + TYPE ICEBERG, + TOKEN '⟨r2_token⟩' +); + +``` + +You can create a token by following the [create an API token](https://developers.cloudflare.com/r2/data-catalog/get-started/#3-create-an-api-token) steps in getting started. + +Then, attach the catalog with the following commands. + +```sql +ATTACH '⟨warehouse⟩' AS my_r2_catalog ( + TYPE ICEBERG, + ENDPOINT '⟨catalog-uri⟩' +); +``` + +The variables for `warehouse` and `catalog-uri` will be available under the settings of the desired R2 Object Storage Catalog (R2 Object Store > Catalog name > Settings). + +### Polaris + +To attach to a [Polaris](https://polaris.apache.org) catalog the following commands will work. + +```sql +CREATE SECRET polaris_secret ( + TYPE ICEBERG, + CLIENT_ID '⟨admin⟩', + CLIENT_SECRET '⟨password⟩', +); +``` + +```sql +ATTACH 'quickstart_catalog' as polaris_catalog ( + TYPE ICEBERG, + ENDPOINT '⟨polaris_rest_catalog_endpoint⟩' +); +``` + + +### Lakekeeper + +To attach to a [Lakekeeper](https://docs.lakekeeper.io) catalog the following commands will work. + +```sql +CREATE SECRET lakekeeper_secret ( + TYPE ICEBERG, + CLIENT_ID '⟨admin⟩', + CLIENT_SECRET '⟨password⟩', + OAUTH2_SCOPE '⟨scope⟩', + OAUTH2_SERVER_URI '⟨lakekeeper_oauth_url⟩' +); +``` + +```sql +ATTACH '⟨warehouse⟩' as lakekeeper_catalog ( + TYPE ICEBERG, + ENDPOINT '⟨lakekeeper_irc_url⟩', + SECRET lakekeeper_secret +); +``` + +## Limitations + +Reading from Iceberg REST Catalogs backed by remote storage that is not S3 or S3Tables is not yet supported. \ No newline at end of file diff --git a/docs/stable/core_extensions/iceberg/overview.md b/docs/stable/core_extensions/iceberg/overview.md index fff433da9b1..476b54a1f18 100644 --- a/docs/stable/core_extensions/iceberg/overview.md +++ b/docs/stable/core_extensions/iceberg/overview.md @@ -5,7 +5,7 @@ redirect_from: null title: Iceberg Extension --- -The `iceberg` extension implements support for the [Apache Iceberg open table format](https://iceberg.apache.org/). +The `iceberg` extension implements support for the [Apache Iceberg open table format](https://iceberg.apache.org/) and can connect to Iceberg REST Catalogs. For information on how to connect to an Iceberg REST Catalog, please see the [Iceberg REST Catalogs]({% link docs/stable/core_extensions/iceberg/iceberg_rest_catalogs.md %}) page. ## Installing and Loading @@ -57,7 +57,7 @@ FROM iceberg_scan('data/iceberg/lineitem_iceberg', allow_moved_paths = true); |-------------:| | 51793 | -> The `allow_moved_paths` option ensures that some path resolution is performed, +> The `allow_moved_paths` option ensures that some path resolution is performed, > which allows scanning Iceberg tables that are moved. You can also address specify the current manifest directly in the query, this may be resolved from the catalog prior to the query, in this example the manifest version is a UUID. @@ -88,6 +88,13 @@ SELECT * FROM iceberg_metadata('data/iceberg/lineitem_iceberg', allow_moved_paths = true); ``` +You can also run the `iceberg_metadata` function on Iceberg tables attached via the REST Catalog: + +```sql +SELECT * +FROM iceberg_metadata(iceberg_table); +``` +
| manifest_path | manifest_sequence_number | manifest_content | status | content | file_path | file_format | record_count | @@ -104,6 +111,13 @@ SELECT * FROM iceberg_snapshots('data/iceberg/lineitem_iceberg'); ``` +You can also run the `iceberg_snapshots` function on Iceberg tables attached via the REST Catalog: + +```sql +SELECT * +FROM iceberg_snapshots(iceberg_table); +``` + | sequence_number | snapshot_id | timestamp_ms | manifest_list |