Skip to content

Add BigLake support and better encoding/decoding of prefix's in catalog response#687

Merged
Tmonster merged 11 commits intomainfrom
add_biglake_support
Feb 8, 2026
Merged

Add BigLake support and better encoding/decoding of prefix's in catalog response#687
Tmonster merged 11 commits intomainfrom
add_biglake_support

Conversation

@Tmonster
Copy link
Copy Markdown
Collaborator

@Tmonster Tmonster commented Feb 5, 2026

This builds off of #674

We detect if a prefix is already encoded by URLDecoding it, if it equals the prefix returned by the catalog, we assume it is not URL encoded, and each / is a separate component. If it does not equal the prefix returned by the catalog, we assume it is encoded and that the prefix is meant to be one component in the URL.

S3Tables expects 1 component
Glue can be either 1 component or multiple components

Here is a breakdown of some current catalogs

catalog prefix example encoded is multiple components?
Biglake projects/1057666841514/catalogs/biglake-public-nyc-taxi-iceberg no yes
Glue catalogs/<account_id>:s3tablescatalog:<bucket_name> no yes&no
s3tables arn%3Aaws%3As3tables%3Aus-east-1%3A<account_id>%3Abucket%2F<bucket_name> yes no

@Tmonster
Copy link
Copy Markdown
Collaborator Author

Tmonster commented Feb 5, 2026

This PR contains a fix for polaris

Copy link
Copy Markdown
Collaborator

@Tishj Tishj left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks LGTM 👍

@Tmonster Tmonster merged commit 09440dd into main Feb 8, 2026
78 of 84 checks passed
@Tmonster Tmonster deleted the add_biglake_support branch February 9, 2026 11:37
@talatuyarer
Copy link
Copy Markdown
Contributor

Thank you @Tmonster and @Tishj

Where do we need to update for documentation ?

With this change Biglake catalog's table will run with GCS also too. But in documentation it says it does not support any other blob storage except S3. https://duckdb.org/docs/stable/core_extensions/iceberg/iceberg_rest_catalogs#:~:text=lakekeeper_secret%27%0A)%3B-,Limitations,that%20is%20not%20S3%20or%20S3%20Tables%20is%20not%20yet%20supported.,-About%20this%20page

And also I would like to add a Biglake example that i share on the documentation. #665

@Tmonster
Copy link
Copy Markdown
Collaborator Author

@talatuyarer
I have an internal issue to add to the docs, but if you want to take a stab at it feel free.
Here is a PR I opened to add our current documentation, that should be enough to get started

duckdb/duckdb-web#5410

@astronautas
Copy link
Copy Markdown

astronautas commented Feb 11, 2026

This is not released as far as I understand? Just tested:

_duckdb.BinderException: Binder Error: Unknown parameter 'extra_http_headers' for secret type 'iceberg' with default provider 'config'

@Tmonster
Copy link
Copy Markdown
Collaborator Author

No, it is not available yet. It will be available in the v1.5.0 release, however, which is planned for sometime later february

@talatuyarer
Copy link
Copy Markdown
Contributor

is there any nightly build for preview that we can use without building code ? @Tmonster

@Tmonster
Copy link
Copy Markdown
Collaborator Author

Tmonster commented Feb 11, 2026

Unfortunately no. Building locally is the only option right now. It would require a nightly DuckDB build, and there is no guarantee that extensions are built for every nightly DuckDB build

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants