Add BigLake support and better encoding/decoding of prefix's in catalog response#687
Add BigLake support and better encoding/decoding of prefix's in catalog response#687
Conversation
Enhanced `IRCEndpointBuilder::AddPathComponent` to split path components by slashes to prevent slashes get encoded.
…in IRCEndpointBuilder. Update tests to verify behavior with empty and multiple custom headers.
…nsistency in path handling
|
This PR contains a fix for polaris |
|
Thank you @Tmonster and @Tishj Where do we need to update for documentation ? With this change Biglake catalog's table will run with GCS also too. But in documentation it says it does not support any other blob storage except S3. https://duckdb.org/docs/stable/core_extensions/iceberg/iceberg_rest_catalogs#:~:text=lakekeeper_secret%27%0A)%3B-,Limitations,that%20is%20not%20S3%20or%20S3%20Tables%20is%20not%20yet%20supported.,-About%20this%20page And also I would like to add a Biglake example that i share on the documentation. #665 |
|
@talatuyarer |
|
This is not released as far as I understand? Just tested: |
|
No, it is not available yet. It will be available in the v1.5.0 release, however, which is planned for sometime later february |
|
is there any nightly build for preview that we can use without building code ? @Tmonster |
|
Unfortunately no. Building locally is the only option right now. It would require a nightly DuckDB build, and there is no guarantee that extensions are built for every nightly DuckDB build |
This builds off of #674
We detect if a prefix is already encoded by URLDecoding it, if it equals the prefix returned by the catalog, we assume it is not URL encoded, and each
/is a separate component. If it does not equal the prefix returned by the catalog, we assume it is encoded and that the prefix is meant to be one component in the URL.S3Tables expects 1 component
Glue can be either 1 component or multiple components
Here is a breakdown of some current catalogs