Add register_table procedure support for iceberg using existing metadata#14325
Add register_table procedure support for iceberg using existing metadata#14325krvikash wants to merge 0 commit intotrinodb:masterfrom
Conversation
|
Added empty commit to retrigger the build after #14323 merged. |
There was a problem hiding this comment.
Move the constants to IcebergUtil and make them private.
There was a problem hiding this comment.
Point out to the user which are the duplicated metadata files in such a situation.
There was a problem hiding this comment.
Probably ICEBERG_INVALID_METADATA is the right error code in such a situation.
There was a problem hiding this comment.
Remove the javadoc about the params and return. Given the description of the method, these aspects are obvious.
There was a problem hiding this comment.
Use Optional<String> for metadataFilename.
There was a problem hiding this comment.
Add a comment stating that this is done in the same fashion as in Iceberg.
I found the method in BaseMetastoreTableOperations#parseVersion.
There was a problem hiding this comment.
Do we need to verify whether the schema in which we want to register the table exists?
Do we need to verify if a table with the same name already exists in the specified schema?
There was a problem hiding this comment.
Good catch. Yes both the points need to be checked. I will make the changes.
There was a problem hiding this comment.
The error code is not appropriate. The file system is fine, is just that the file at the location provided is missing.
|
Let's add some tests. Note that The metadata files contain full path names for the You can find inspiration on how to get access within the integration tests to the metastore / aws client directly by looking at these tests:
|
f8e951d to
8f65f29
Compare
8f65f29 to
bc4e79a
Compare
|
This PR got closed because of my mistake during running GitHub commands. Reopen a new PR #14375 @findinpath, Addressed your review comments in #14375 |
For next time, you can reopen the same PR. There's a button at the bottom of the page |
Description
Adding
register_tableprocedure for iceberg connector to register table using existing metadata..metadata.jsoninside themetadatafolder and use it for creating the tableschema_name,table_name, andtable_locationshould be not-null and valid, Otherwise, the exception will be thrownmetadata_file_location(See the usage below)Valid usages:
register_table(schema_name => ..., table_name => ..., table_location => ...)register_table(schema_name => ..., table_name => ..., table_location => ..., metadata_file_location => ...)Sample Queries:
CALL iceberg.system.register_table('default', 'src_22', 'hdfs://hadoop-master:9000/user/hive/warehouse/orders_5-581fad8517934af6be1857a903559d44');CALL iceberg.system.register_table('default', 'src_22', 'hdfs://hadoop-master:9000/user/hive/warehouse/orders_5-581fad8517934af6be1857a903559d44', null);CALL iceberg.system.register_table('default', 'src_22', 'hdfs://hadoop-master:9000/user/hive/warehouse/orders_5-581fad8517934af6be1857a903559d44', '00003-409702ba-4735-4645-8f14-09537cc0b2c8.metadata.json');TODO:
Add test cases
Non-technical explanation
NA
Release notes
( ) This is not user-visible or docs only and no release notes are required.
(X) Release notes are required, please propose a release note for me.
( ) Release notes are required, with the following suggested text: