Spark: Support truncate in FunctionCatalog #5431
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
kbendick
force-pushed
the
kb-add-spark-truncate-function
branch
5 times, most recently
from
a602e50 to
eae6bb8
Compare
rdblue
changed the title
rdblue
reviewed
Aug 4, 2022
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/functions/TruncateFunction.java
Outdated
Show resolved
Hide resolved
rdblue
reviewed
Aug 4, 2022
rdblue
reviewed
Aug 4, 2022
core/src/test/java/org/apache/iceberg/util/TestTruncateUtil.java
Outdated
Show resolved
Hide resolved
rdblue
reviewed
Aug 4, 2022
core/src/test/java/org/apache/iceberg/util/TestTruncateUtil.java
Outdated
Show resolved
Hide resolved
rdblue
reviewed
Aug 4, 2022
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/functions/TruncateFunction.java
Outdated
Show resolved
Hide resolved
rdblue
reviewed
Aug 4, 2022
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/functions/TruncateFunction.java
Outdated
Show resolved
Hide resolved
rdblue
reviewed
Aug 4, 2022
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/functions/TruncateFunction.java
Outdated
Show resolved
Hide resolved
rdblue
reviewed
Aug 4, 2022
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/functions/TruncateFunction.java
Outdated
Show resolved
Hide resolved
rdblue
reviewed
Aug 4, 2022
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/functions/TruncateFunction.java
Outdated
Show resolved
Hide resolved
rdblue
reviewed
Aug 4, 2022
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/functions/TruncateFunction.java
Outdated
Show resolved
Hide resolved
rdblue
reviewed
Aug 4, 2022
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/functions/TruncateFunction.java
Outdated
Show resolved
Hide resolved
rdblue
reviewed
Aug 4, 2022
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/functions/TruncateFunction.java
Outdated
Show resolved
Hide resolved
rdblue
reviewed
Aug 4, 2022
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/functions/TruncateFunction.java
Outdated
Show resolved
Hide resolved
rdblue
reviewed
Aug 4, 2022
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/functions/TruncateFunction.java
Outdated
Show resolved
Hide resolved
rdblue
reviewed
Aug 4, 2022
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/functions/TruncateFunction.java
Outdated
Show resolved
Hide resolved
rdblue
reviewed
Aug 4, 2022
spark/v3.3/spark/src/test/java/org/apache/iceberg/spark/sql/TestSparkTruncateFunction.java
Outdated
Show resolved
Hide resolved
rdblue
reviewed
Aug 4, 2022
spark/v3.3/spark/src/test/java/org/apache/iceberg/spark/sql/TestSparkTruncateFunction.java
Outdated
Show resolved
Hide resolved
rdblue
reviewed
Aug 4, 2022
spark/v3.3/spark/src/test/java/org/apache/iceberg/spark/sql/TestSparkTruncateFunction.java
Outdated
Show resolved
Hide resolved
rdblue
reviewed
Aug 4, 2022
spark/v3.3/spark/src/test/java/org/apache/iceberg/spark/sql/TestSparkTruncateFunction.java
Outdated
Show resolved
Hide resolved
rdblue
reviewed
Aug 4, 2022
spark/v3.3/spark/src/test/java/org/apache/iceberg/spark/sql/TestSparkTruncateFunction.java
Outdated
Show resolved
Hide resolved
rdblue
reviewed
Aug 4, 2022
spark/v3.3/spark/src/test/java/org/apache/iceberg/spark/sql/TestSparkTruncateFunction.java
Outdated
Show resolved
Hide resolved
…ll input values in invoke
… on null on the special accesssors of certain row types that might not tolerate it
kbendick
force-pushed
the
kb-add-spark-truncate-function
branch
from
ddb851c to
f33b356
Compare
kbendick
force-pushed
the
kb-add-spark-truncate-function
branch
from
1d63a04 to
9e86776
Compare
…ove separate function for validation
…r fewer than 2 and more than 2 arguments
…ccepted in storage partitioned joins impl
Contributor
Author
|
@aokolnychyi @rdblue I addressed all of your feedback. Please take a look when you get a chance. Also, on the subject of nullability for the Knowing that Spark isn't the best at keeping track of nullability, I think this is better and adheres to the contract laid out in But take a look at the commit that added nullability-checking on the width field if you'd like: ad5343f |
kbendick
force-pushed
the
kb-add-spark-truncate-function
branch
from
fd7ef4b to
ce6dba1
Compare
rdblue
approved these changes
Aug 11, 2022
Contributor
|
Looks good to me. @aokolnychyi, do you want to take another look? |
Contributor
|
@rdblue, let me take a quick now. |
Contributor
|
Thanks, @kbendick! Could you cherry-pick this to 3.2? |
Contributor
Author
|
Here's the PR for bucket. All the feedback from this PR was more or less applied there as well - #5513 |
Contributor
Author
Sure thing! |
zhongyujiang
pushed a commit
to zhongyujiang/iceberg
that referenced
this pull request
Apr 16, 2025
(cherry picked from commit 6a5051b)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is an offshoot of #5305 and partially closes #5349.
Adds a
system.truncatefunction that can be used in Spark SQL, as well as can be used as aFunctionCatalogfunction that can be turned into a transform for storage partitioned joins.This also breaks the definition of
Truncateout into utility functions inside ofTruncateUtil. Because different usages validate input at different times, all of the functions inTruncateUtildo not validate their input and instead assume that the input is validated by the calling code. This allows for theTruncatetransforms to validate their width one time (on instantiation), and for the Sparktruncatefunction to skip input validation for faster generated code.