Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: iceberg table engine. #13835

Merged
merged 6 commits into from
Dec 4, 2023
Merged

feat: iceberg table engine. #13835

merged 6 commits into from
Dec 4, 2023

Conversation

youngsofun
Copy link
Member

@youngsofun youngsofun commented Nov 28, 2023

I hereby agree to the terms of the CLA available at: https://databend.rs/dev/policies/cla/

Summary

part of #13787

use table options

create table <name> engine = iceberg location=<location> [connection_name =<connection_name>]


This change is Reviewable

@github-actions github-actions bot added the pr-feature this PR introduces a new feature to the codebase label Nov 28, 2023
@youngsofun youngsofun force-pushed the ice branch 2 times, most recently from 4babfbd to 7e578f0 Compare November 28, 2023 14:27
@youngsofun
Copy link
Member Author

youngsofun commented Nov 29, 2023

another issue:

in current impl of this pr.
url + connenction is translated to storage_parameter.
if url is a stage (object), or connection contain named connection object.
the location for table should change with the stage/connection object.

so I now prefer use table options:

create table <name> engine = iceberg location=<location> connection=<connection_name>
cc @Xuanwo @BohuTANG

acturally the gramma for ending <url> [connection=] is a little strange to me.

@youngsofun
Copy link
Member Author

another issue:

in current impl of this pr.

url + connenction is translated to storage_parameter.

if url is a stage (object), or connection contain named connection object.

the location for table should change with the stage/connection object.

so I now prefer use table options:

create table <name> engine = iceberg location=<location> connection=<connection_name>

cc @Xuanwo @BohuTANG

acturally the gramma for ending <url> [connection=] is a little strange to me.

in this way, we share the same grammar for options with table fuctions like inferschema

@youngsofun youngsofun force-pushed the ice branch 3 times, most recently from 6c67a18 to 5901f5d Compare December 4, 2023 07:58
@youngsofun
Copy link
Member Author

youngsofun commented Dec 4, 2023

https://github.com/datafuselabs/databend/actions/runs/7083601731/job/19276276688?pr=13835

  ==> installing ziglang...
  Installing zig.
  Error: Not tolerating missing names on install, stopping.
  Error: The process '/usr/local/bin/docker' failed with exit code 1
      at ExecState._setResult (/runner/_work/_actions/PyO3/maturin-action/v1/dist/index.js:1702:25)
      at ExecState.CheckComplete (/runner/_work/_actions/PyO3/maturin-action/v1/dist/index.js:1685:18)
      at ChildProcess.<anonymous> (/runner/_work/_actions/PyO3/maturin-action/v1/dist/index.js:1579:27)
      at ChildProcess.emit (node:events:513:28)
      at maybeClose (node:internal/child_process:1100:16)
      at Process.ChildProcess._handle.onexit (node:internal/child_process:304:5)
  Error: The process '/usr/local/bin/docker' failed with exit code 1

cc @everpcpc

@youngsofun
Copy link
Member Author

@Xuanwo @sundy-li

please review again

@youngsofun youngsofun added this pull request to the merge queue Dec 4, 2023
@BohuTANG BohuTANG removed this pull request from the merge queue due to a manual request Dec 4, 2023
@BohuTANG BohuTANG merged commit 6bc7124 into databendlabs:main Dec 4, 2023
68 checks passed
@youngsofun youngsofun deleted the ice branch December 4, 2023 23:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
pr-feature this PR introduces a new feature to the codebase
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants