Skip to content

Conversation

@zeroshade
Copy link
Member

fixes #407

Update and improve the Glue catalog CreateTable to properly set the parameters on the table itself and the columns (such as iceberg.field.id and iceberg.field.optional).

Also, write out the metadata first and then pass the location to Glue as part of creating the table.

Copy link
Member Author

@zeroshade zeroshade left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lliangyu-lin can you take a look and confirm if this fixes the issue you filed?

@zeroshade zeroshade force-pushed the glue-catalog-create branch from 128ae7c to 55c2b45 Compare April 29, 2025 20:21
@lliangyu-lin
Copy link
Contributor

@zeroshade Thank you for working on this! The implementation looks good to me.

But somehow when I test locally with actual AWS Credentials, I'm getting 400 from Glue InvalidInputException: Cannot use reserved parameters table_type or metadata_location while creating an iceberg table. I checked other iceberg client implementations and they did use table_type and metadata_location as well. So let me do some check and get back to you on this.

@zeroshade zeroshade force-pushed the glue-catalog-create branch from 33f0375 to 4c8d78a Compare May 1, 2025 16:25
Copy link
Contributor

@lliangyu-lin lliangyu-lin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Thanks @zeroshade

Copy link
Contributor

@Fokko Fokko left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks great @zeroshade Thanks!


icebergFieldIDKey = "iceberg.field.id"
icebergFieldOptionalKey = "iceberg.field.optional"
icebergFieldCurrentKey = "iceberg.field.current"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I forgot about this one. Will it ever be false? 🤔 The current schema should be stored in Glue.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interesting: apache/iceberg#3888

@Fokko Fokko merged commit 3398683 into apache:main May 3, 2025
10 checks passed
@Fokko
Copy link
Contributor

Fokko commented May 3, 2025

Thanks for working on this @zeroshade and thanks for the review @lliangyu-lin 🙌

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Glue catalog create table doesn't work with schema with required columns

3 participants