-
Notifications
You must be signed in to change notification settings - Fork 2.9k
Core: Validate transforms while building partition type #2992
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
kbendick
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This looks good to me overall. A few questions for my own understanding.
A few questions:
- I'm assuming you're using this subset of metadata tables because they're partitioned?
- Is there a way to add a test against a table itself (and not just the metadata tables)?
yyanyy
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I guess I'm not familiar with the context, is this case only going to be hit when there's a new writer that commit partition spec to the table that contains a new transform, and the metadata tables associated with the table is read by the old reader which does not recognize this transform; and we are updating the logic for this old reader to fail gracefully?
|
@yyanyy, that's exactly the use case. |
jackye1995
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this also looks good to me
|
Thanks for reviewing, @kbendick @yyanyy @jackye1995! |
Until we store the output type of transforms, we should fail building the partition type if we hit
UnknownTransform.