Add void transform that always produces null #924
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
Author
Comment on lines
+64
to
+71
| public String toHumanString(Void value) { | ||
| return "null"; | ||
| } | ||
|
|
||
| @Override | ||
| public String toString() { | ||
| return "void"; | ||
| } |
Contributor
There was a problem hiding this comment.
Minor nit here to think about . . . we seem to be using both void and null. Would it make more sense to just consistently use void as it seems to better indicate there there is no expected value?
Contributor
Author
There was a problem hiding this comment.
Null is the human-readable string for the value produced by the transform. Void is the name of the transform. I considered naming it something like always_null but void seemed shorter and less error prone (was that alwaysNull or always-null?)
Contributor
|
One minor comment, but +1 (pending checks) |
rdblue
force-pushed
the
add-null-constant-transform
branch
from
659fcd4 to
688a37f
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This adds a new transform function,
void, that always produces a null value. Becausevoidandnullare Java keywords, thePartitionSpecBuilderis configured usingalwaysNull.The purpose of this transform is to be a stand-in for partition transforms that are removed from a spec. In the v1 table format, IDs for partition fields are not tracked by
PartitionSpec. Instead, they are assigned starting at 1000 for each spec. Because tables may have more than one spec, manifest files could have incompatible partition field structs. This is not a problem for job planning because each manifest is read independently, but it can break metadata tables that show a union of all manifest data files or entries.The
voidtransform can be used to avoid a problem with ID assignment. If a table has two partition fields,1000: categorical string, 1001: ts_day int, then removing thecategoricalpartition will create a new partition spec with1000: ts_day int. That would create a problem in the metadata tables. Instead of deleting the categorical partition, it should be replaced with avoidpartition to keep the IDs aligned:1000: always_null string, 1001: ts_day int.