Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow additional options when creating an external table #4125

Closed
timvw opened this issue Nov 7, 2022 · 3 comments · Fixed by #4126
Closed

Allow additional options when creating an external table #4125

timvw opened this issue Nov 7, 2022 · 3 comments · Fixed by #4126
Labels
enhancement New feature or request

Comments

@timvw
Copy link
Contributor

timvw commented Nov 7, 2022

Is your feature request related to a problem or challenge? Please describe what you are trying to do.
CREATE EXTERNAL TABLE should allow for custom (table type specific options).

Eg: For CSV the has_header and delimiter option are available but it's not possible to add custom options but it's not possible to specify the timestamp for loading a delta/iceberg table.

In spark this is possible via the generic OPTIONS parameter.

Describe the solution you'd like
Extend datafusion such that these options can be provided

CREATE EXTERNAL TABLE magic 
STORED AS DELTALAKE 
OPTIONS ( at='1667827882' )
LOCATION 's3://mybucket/magic'

Describe alternatives you've considered
We could not support this

Additional context
Relates to #3867

@avantgardnerio
Copy link
Contributor

@timvw I think this is an excellent idea. Syntactically it's in line with postgres' create external table and I think it allows for run-time extensibility that is required if we want to have a healthy ecosystem of 3rd party TableProviders .

@timvw
Copy link
Contributor Author

timvw commented Nov 7, 2022

Ok, currently the syntax is slightly different, but happy to change that ;)

current: [ OPTIONS ( option='value' [, ... ] ) ]
postgres: [ OPTIONS ( option 'value' [, ... ] ) ]

@timvw
Copy link
Contributor Author

timvw commented Nov 7, 2022

Updated the code to be the same as postgres..

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants