-
Notifications
You must be signed in to change notification settings - Fork 3k
Backport Parquet encoding tests for Spark 3.5 #13859
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
|
Thanks for the PR! For context, we're trying to make sure spark 3.5 and 4.0 have feature parity in the 1.10 release since this is the first release with Spark 4.0. More context in https://lists.apache.org/thread/8xzbg1wqft2grv8v1f13vb86vd8f7rjd |
...c/test/java/org/apache/iceberg/spark/data/vectorized/parquet/TestParquetVectorizedReads.java
Show resolved
Hide resolved
...c/test/java/org/apache/iceberg/spark/data/vectorized/parquet/TestParquetVectorizedReads.java
Show resolved
Hide resolved
|
@eric-maynard Thanks for the back-port PR! Should we put these golden files in a common place so we don't have to add them for each Spark version? |
|
@RussellSpitzer @stevenzwu @nastra @kevinjqliu |
|
How about |
|
I removed this one from the 1.10.0 milestone as it is just test as @kevinjqliu says |
|
This is actually not just a test — the actual reader is being backported
too.
…On Thu, Aug 28, 2025 at 7:02 PM Steven Zhen Wu ***@***.***> wrote:
*stevenzwu* left a comment (apache/iceberg#13859)
<#13859 (comment)>
I removed this one from the 1.10.0 milestone as it is just test as
@kevinjqliu <https://github.com/kevinjqliu> says
—
Reply to this email directly, view it on GitHub
<#13859 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AFRE3SF6TYD2F7LXPUHAVAT3P6YFBAVCNFSM6AAAAACEFZU5WCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTEMZVGQ3DGOBSG4>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
@eric-maynard from the 3 commits listed in the description (c3d50e1, e667670, and 5d5e0a3), i see only test files in the spark/v4.0 folder do you know where i can find the reader being backported? |
|
Hey @kevinjqliu, my mistake, this is really only porting the tests. So you're right that it should not be a blocker. |
...c/test/java/org/apache/iceberg/spark/data/vectorized/parquet/TestParquetVectorizedReads.java
Show resolved
Hide resolved
...c/test/java/org/apache/iceberg/spark/data/vectorized/parquet/TestParquetVectorizedReads.java
Show resolved
Hide resolved
...c/test/java/org/apache/iceberg/spark/data/vectorized/parquet/TestParquetVectorizedReads.java
Outdated
Show resolved
Hide resolved
...c/test/java/org/apache/iceberg/spark/data/vectorized/parquet/TestParquetVectorizedReads.java
Show resolved
Hide resolved
...c/test/java/org/apache/iceberg/spark/data/vectorized/parquet/TestParquetVectorizedReads.java
Outdated
Show resolved
Hide resolved
stevenzwu
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM. just a nit comment
...c/test/java/org/apache/iceberg/spark/data/vectorized/parquet/TestParquetVectorizedReads.java
Outdated
Show resolved
Hide resolved
...c/test/java/org/apache/iceberg/spark/data/vectorized/parquet/TestParquetVectorizedReads.java
Outdated
Show resolved
Hide resolved
...c/test/java/org/apache/iceberg/spark/data/vectorized/parquet/TestParquetVectorizedReads.java
Outdated
Show resolved
Hide resolved
|
Thanks @eric-maynard for the contribution, @nastra @huaxingao @kevinjqliu for the review |
This is a backport of c3d50e1, e667670, and 5d5e0a3, where tests were added for Spark 4.0 but not Spark 3.5.