-
Notifications
You must be signed in to change notification settings - Fork 2.9k
[ISSUE #855] Add the Test Util And ORC Writer for Spark #857
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@shawnding , just left some comments for you. I think it would be better if you could run ./gradlew build before submitting your PR next time.
spark/src/test/java/org/apache/iceberg/spark/source/NestedRecord.java
Outdated
Show resolved
Hide resolved
spark/src/test/java/org/apache/iceberg/spark/source/SimpleRecord.java
Outdated
Show resolved
Hide resolved
spark/src/test/java/org/apache/iceberg/spark/source/TestORCWrite.java
Outdated
Show resolved
Hide resolved
spark/src/test/java/org/apache/iceberg/spark/source/TestORCWrite.java
Outdated
Show resolved
Hide resolved
spark/src/test/java/org/apache/iceberg/spark/source/TestORCWrite.java
Outdated
Show resolved
Hide resolved
spark/src/test/java/org/apache/iceberg/spark/source/TestORCWrite.java
Outdated
Show resolved
Hide resolved
spark/src/test/java/org/apache/iceberg/spark/source/NestedRecord.java
Outdated
Show resolved
Hide resolved
spark/src/test/java/org/apache/iceberg/spark/source/NestedRecord.java
Outdated
Show resolved
Hide resolved
spark/src/test/java/org/apache/iceberg/spark/source/NestedRecord.java
Outdated
Show resolved
Hide resolved
spark/src/test/java/org/apache/iceberg/spark/source/TestORCWrite.java
Outdated
Show resolved
Hide resolved
223ce04 to
8e0aaea
Compare
|
HI @rdsr , would you mind to take another look or give some advices on next steps? thanks |
|
Thanks @shawnding . I'll try to review this today! |
spark/src/main/java/org/apache/iceberg/spark/source/Writer.java
Outdated
Show resolved
Hide resolved
spark/src/test/java/org/apache/iceberg/spark/data/TestSparkOrcReader.java
Outdated
Show resolved
Hide resolved
spark/src/test/java/org/apache/iceberg/spark/source/TestSparkDataWrite.java
Outdated
Show resolved
Hide resolved
spark/src/test/java/org/apache/iceberg/spark/source/TestSparkDataWrite.java
Show resolved
Hide resolved
|
This looks good to me, except for two things:
|
|
One more thing that I thought of: if this is adding the ability to write ORC files, then we should make sure all of the parameterized Spark tests that currently test Parquet and Avro are also applied to ORC. |
Sorry for the miscommunication: I'm asking you to remove the more complicated nested type from this PR. Let's get the existing tests running on ORC for this PR and then we can add more test cases in a separate one. |
|
Thanks @rdblue , I was removed the nested type.would you mind to take another look or give some advices on next steps? |
|
Looks good to me. Thanks for adding this, @shawnding! |
Add the Test Util And ORC file writer.
Lack of the metrics information and file length check.