Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions _data/menu_docs_stable.json
Original file line number Diff line number Diff line change
Expand Up @@ -1082,6 +1082,10 @@
"page": "Overview",
"url": "overview"
},
{
"page": "Iceberg REST Catalogs",
"url": "iceberg_rest_catalogs"
},
{
"page": "Amazon S3 Tables",
"url": "amazon_s3_tables"
Expand Down
15 changes: 1 addition & 14 deletions docs/stable/core_extensions/iceberg/amazon_s3_tables.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,21 +4,8 @@ redirect_from: null
title: Amazon S3 Tables
---

The `iceberg` extension supports reading Iceberg tables stored in [Amazon S3 Tables](https://aws.amazon.com/s3/features/tables/).
The `iceberg` extension supports reading Iceberg tables stored in [Amazon S3 Tables](https://aws.amazon.com/s3/features/tables/). Make sure you have the Iceberg extension installed by following the steps at [Installing and Loading]({% link docs/stable/core_extensions/iceberg/overview.md %})

## Requirements

The S3 Tables support is currently experimental.
To use it, install the following extensions:

```sql
FORCE INSTALL aws FROM core_nightly;
FORCE INSTALL httpfs FROM core_nightly;
FORCE INSTALL iceberg FROM core_nightly;
```

> If you want to switch back to using extensions from the `core` repository,
> follow the [extension documentation]({% link docs/stable/extensions/installing_extensions.md %}#force-installing-to-upgrade-extensions).

## Connecting to Amazon S3 Tables

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,21 +4,8 @@ redirect_from: null
title: Amazon SageMaker Lakehouse (AWS Glue)
---

The `iceberg` extension supports reading Iceberg tables through the [Amazon SageMaker Lakehouse (a.k.a. AWS Glue)](https://aws.amazon.com/sagemaker/lakehouse/) catalog.
The `iceberg` extension supports reading Iceberg tables through the [Amazon SageMaker Lakehouse (a.k.a. AWS Glue)](https://aws.amazon.com/sagemaker/lakehouse/) catalog. Make sure you have the Iceberg extension installed by following the steps at [Installing and Loading]({% link docs/stable/core_extensions/iceberg/overview.md %})

## Requirements

The S3 Tables support is currently experimental.
To use it, install the following extensions:

```sql
FORCE INSTALL aws FROM core_nightly;
FORCE INSTALL httpfs FROM core_nightly;
FORCE INSTALL iceberg FROM core_nightly;
```

> If you want to switch back to using extensions from the `core` repository,
> follow the [extension documentation]({% link docs/stable/extensions/installing_extensions.md %}#force-installing-to-upgrade-extensions).

## Connecting to Amazon SageMaker Lakehouse

Expand Down
144 changes: 144 additions & 0 deletions docs/stable/core_extensions/iceberg/iceberg_rest_catalogs.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,144 @@
---
layout: docu
redirect_from: null # maybe redirect from old amazon_s3_tables and amazon_sagemaker_lakehouse docs?
title: Iceberg REST Catalogs
---

The `iceberg` extension supports attaching Iceberg REST Catalogs. Before attaching an Iceberg REST Catalog, you must install the `iceberg` extension by following the instructions located in the [overview]({% link docs/stable/core_extensions/iceberg/overview.md %}).

If you are attaching to an Iceberg REST Catalog managed by Amazon, please see the instructions for attaching to [Amazon S3 tables]({% link docs/stable/core_extensions/iceberg/amazon_s3_tables.md %}) or [Amazon Sagemaker Lakehouse]({% link docs/stable/core_extensions/iceberg/amazon_sagemaker_lakehouse.md %}).

For all other Iceberg REST Catalogs, you can follow the instructions below. Please see the [Examples](#examples) section for questions about specific catalogs.

Most Iceberg REST Catalogs authenticate via OAuth2. You can use the existing DuckDB secret workflow to store login credentials for the OAuth2 service.

```sql
CREATE SECRET iceberg_secret (
TYPE ICEBERG,
CLIENT_ID '⟨admin⟩',
CLIENT_SECRET '⟨password⟩',
OAUTH2_SERVER_URI '⟨http://irc_host_url.com/v1/oauth/tokens⟩'
);
```

If you already have a Bearer token, you can pass it directly to your `CREATE SECRET` statement

```sql
CREATE SECRET iceberg_secret (
TYPE ICEBERG,
TOKEN '⟨bearer_token⟩'
);
```

You can attach the Iceberg catalog with the following [`ATTACH`]({% link docs/stable/sql/statements/attach.md %}) statement.

```sql
ATTACH '⟨warehouse⟩' AS iceberg_catalog (
TYPE iceberg,
SECRET iceberg_secret, -- pass a specific secret name to prevent ambiguity
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not really for ambiguity, I think?
It's more that authorization options can be provided directly inside the ATTACH (in the same way as the "create secret" options), but for reusability/deduplication the secret can be made beforehand and reused in the attach afterwards

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also that, but I can imagine users attach multiple catalogs that all require a different secret. I think it's best to go over all the options in a separate section

ENDPOINT ⟨https://rest_endpoint.com⟩
);
```

To see the available tables run
```sql
SHOW ALL TABLES;
```

### ATTACH OPTIONS

A REST Catalog with OAuth2 authorization can also be attached with just an `ATTACH` statement. See the complete list of `ATTACH` options for a REST catalog below.


| Parameter | Type | Default | Description |
| ---------------------------- | ---------- | -------- | ---------------------------------------------------------- |
| `ENDPOINT_TYPE` | `VARCHAR` | `NULL` | Used for attaching S3Tables or Glue catalogs. Allowed values are 'GLUE' and 'S3_TABLES' |
| `ENDPOINT` | `VARCHAR` | `NULL` | URL endpoint to communicate with the REST Catalog. Cannot be used in conjunction with `ENDPOINT_TYPE` |
| `SECRET` | `VARCHAR` | `NULL` | Name of secret used to communicate with the REST Catalog |
| `CLIENT_ID` | `VARCHAR` | `NULL` | CLIENT_ID used for Secret |
| `CLIENT_SECRET` | `VARCHAR` | `NULL` | CLIENT_SECRET needed for Secret |
| `DEFAULT_REGION` | `VARCHAR` | `NULL` | A Default region to use when communicating with the storage layer |
| `OAUTH2_SERVER_URI` | `VARCHAR` | `NULL` | OAuth2 server url for getting a Bearer Token |
| `AUTHORIZATION_TYPE` | `VARCHAR` | `OAUTH2` | Pass `SigV4` for Catalogs the require SigV4 authorization |


The following options can only be passed to a `CREATE SECRET` statement, and they require `AUTHORIZATION_TYPE` to be `OAUTH2`

| Parameter | Type | Default | Description |
| ---------------------------- | ---------- | -------- | ---------------------------------------------------------- |
| `OAUTH2_GRANT_TYPE` | `VARCHAR` | `NULL` | Grant Type when requesting an OAuth Token |
| `OAUTH2_SCOPE` | `VARCHAR` | `NULL` | Requested scope for the returned OAuth Access Token |

## Specific Catalog Examples

### R2 Catalog

To attach to an [R2 cloudflare](https://developers.cloudflare.com/r2/data-catalog/) managed catalog follow the attach steps below.


```sql
CREATE SECRET r2_secret (
TYPE ICEBERG,
TOKEN '⟨r2_token⟩'
);

```

You can create a token by following the [create an API token](https://developers.cloudflare.com/r2/data-catalog/get-started/#3-create-an-api-token) steps in getting started.

Then, attach the catalog with the following commands.

```sql
ATTACH '⟨warehouse⟩' AS my_r2_catalog (
TYPE ICEBERG,
ENDPOINT '⟨catalog-uri⟩'
);
```

The variables for `warehouse` and `catalog-uri` will be available under the settings of the desired R2 Object Storage Catalog (R2 Object Store > Catalog name > Settings).

### Polaris

To attach to a [Polaris](https://polaris.apache.org) catalog the following commands will work.

```sql
CREATE SECRET polaris_secret (
TYPE ICEBERG,
CLIENT_ID '⟨admin⟩',
CLIENT_SECRET '⟨password⟩',
);
```

```sql
ATTACH 'quickstart_catalog' as polaris_catalog (
TYPE ICEBERG,
ENDPOINT '⟨polaris_rest_catalog_endpoint⟩'
);
```


### Lakekeeper

To attach to a [Lakekeeper](https://docs.lakekeeper.io) catalog the following commands will work.

```sql
CREATE SECRET lakekeeper_secret (
TYPE ICEBERG,
CLIENT_ID '⟨admin⟩',
CLIENT_SECRET '⟨password⟩',
OAUTH2_SCOPE '⟨scope⟩',
OAUTH2_SERVER_URI '⟨lakekeeper_oauth_url⟩'
);
```

```sql
ATTACH '⟨warehouse⟩' as lakekeeper_catalog (
TYPE ICEBERG,
ENDPOINT '⟨lakekeeper_irc_url⟩',
SECRET lakekeeper_secret
);
```

## Limitations
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: wydt of adding a sentence about catalog refreshing not being supported right now. the work around is reattach

similar to duckdb/duckdb-iceberg#247


Reading from Iceberg REST Catalogs backed by remote storage that is not S3 or S3Tables is not yet supported.
18 changes: 16 additions & 2 deletions docs/stable/core_extensions/iceberg/overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ redirect_from: null
title: Iceberg Extension
---

The `iceberg` extension implements support for the [Apache Iceberg open table format](https://iceberg.apache.org/).
The `iceberg` extension implements support for the [Apache Iceberg open table format](https://iceberg.apache.org/) and can connect to Iceberg REST Catalogs. For information on how to connect to an Iceberg REST Catalog, please see the [Iceberg REST Catalogs]({% link docs/stable/core_extensions/iceberg/iceberg_rest_catalogs.md %}) page.

## Installing and Loading

Expand Down Expand Up @@ -57,7 +57,7 @@ FROM iceberg_scan('data/iceberg/lineitem_iceberg', allow_moved_paths = true);
|-------------:|
| 51793 |

> The `allow_moved_paths` option ensures that some path resolution is performed,
> The `allow_moved_paths` option ensures that some path resolution is performed,
> which allows scanning Iceberg tables that are moved.

You can also address specify the current manifest directly in the query, this may be resolved from the catalog prior to the query, in this example the manifest version is a UUID.
Expand Down Expand Up @@ -88,6 +88,13 @@ SELECT *
FROM iceberg_metadata('data/iceberg/lineitem_iceberg', allow_moved_paths = true);
```

You can also run the `iceberg_metadata` function on Iceberg tables attached via the REST Catalog:

```sql
SELECT *
FROM iceberg_metadata(iceberg_table);
```

<div class="monospace_table"></div>

| manifest_path | manifest_sequence_number | manifest_content | status | content | file_path | file_format | record_count |
Expand All @@ -104,6 +111,13 @@ SELECT *
FROM iceberg_snapshots('data/iceberg/lineitem_iceberg');
```

You can also run the `iceberg_snapshots` function on Iceberg tables attached via the REST Catalog:

```sql
SELECT *
FROM iceberg_snapshots(iceberg_table);
```

<div class="monospace_table"></div>

| sequence_number | snapshot_id | timestamp_ms | manifest_list |
Expand Down
Loading