Add singlenode-hudi product test environment and product tests#14936
Merged
ebyhr merged 1 commit intotrinodb:masterfrom Nov 17, 2022
Merged
Add singlenode-hudi product test environment and product tests#14936ebyhr merged 1 commit intotrinodb:masterfrom
singlenode-hudi product test environment and product tests#14936ebyhr merged 1 commit intotrinodb:masterfrom
Conversation
Contributor
Author
|
@ebyhr @findinpath I have addressed the comments from #14669 and also added a few more tests. Please take a look. |
ebyhr
reviewed
Nov 7, 2022
...auncher/src/main/java/io/trino/tests/product/launcher/env/environment/EnvSinglenodeHudi.java
Outdated
Show resolved
Hide resolved
...auncher/src/main/java/io/trino/tests/product/launcher/env/environment/EnvSinglenodeHudi.java
Outdated
Show resolved
Hide resolved
...uct-tests-launcher/src/main/java/io/trino/tests/product/launcher/suite/suites/SuiteHudi.java
Outdated
Show resolved
Hide resolved
.../main/resources/docker/presto-product-tests/conf/environment/singlenode-hudi/hive.properties
Outdated
Show resolved
Hide resolved
testing/trino-product-tests/src/main/java/io/trino/tests/product/hudi/TestHudiMinioReads.java
Outdated
Show resolved
Hide resolved
testing/trino-product-tests/src/main/java/io/trino/tests/product/hudi/TestHudiMinioReads.java
Outdated
Show resolved
Hide resolved
testing/trino-product-tests/src/main/java/io/trino/tests/product/hudi/TestHudiMinioReads.java
Outdated
Show resolved
Hide resolved
testing/trino-product-tests/src/main/java/io/trino/tests/product/hudi/TestHudiMinioReads.java
Outdated
Show resolved
Hide resolved
testing/trino-product-tests/src/main/java/io/trino/tests/product/hudi/TestHudiMinioReads.java
Outdated
Show resolved
Hide resolved
testing/trino-product-tests/src/main/java/io/trino/tests/product/utils/QueryExecutors.java
Outdated
Show resolved
Hide resolved
ebyhr
reviewed
Nov 8, 2022
Member
There was a problem hiding this comment.
Does Hudi allow specifying partitioned columns whose order is different from the table definition? e.g. CREATE TABLE (c1 int, c2 int, c3 int) PARTITIONED BY (c2, c1). Recently, Delta Lake connector faced correctness issue due to this order difference. It would be nice to have such test if Hudi supports it.
Contributor
Author
There was a problem hiding this comment.
It should be possible. I will check.
codope
force-pushed
the
hudi-product-tests
branch
from
47f7cd4 to
55c9987
Compare
findinpath
reviewed
Nov 11, 2022
...auncher/src/main/java/io/trino/tests/product/launcher/env/environment/EnvSinglenodeHudi.java
Outdated
Show resolved
Hide resolved
...auncher/src/main/java/io/trino/tests/product/launcher/env/environment/EnvSinglenodeHudi.java
Outdated
Show resolved
Hide resolved
codope
force-pushed
the
hudi-product-tests
branch
from
8cec5f3 to
0317040
Compare
findinpath
reviewed
Nov 11, 2022
testing/trino-product-tests/src/main/java/io/trino/tests/product/utils/QueryExecutors.java
Outdated
Show resolved
Hide resolved
findinpath
reviewed
Nov 11, 2022
...rino-product-tests/src/main/java/io/trino/tests/product/hudi/TestHudiSparkCompatibility.java
Outdated
Show resolved
Hide resolved
findinpath
reviewed
Nov 11, 2022
Contributor
There was a problem hiding this comment.
Since we're testing "merge on read" it would be nice to actually have a case where a merge actually applies.
Contributor
Author
There was a problem hiding this comment.
Since, snapshot queries are not yet supported for merge on read tables, so I will just add test for read-optimized queries. We will add snapshot query test in #14786
Contributor
|
The set of product tests even though it is not exhaustive, is a pretty good scaffolding to build upon as we add new features to the hudi connector. Thanks for your contribution. |
codope
force-pushed
the
hudi-product-tests
branch
2 times, most recently
from
e9b91f4 to
4cebd70
Compare
findinpath
reviewed
Nov 14, 2022
...rino-product-tests/src/main/java/io/trino/tests/product/hudi/TestHudiSparkCompatibility.java
Outdated
Show resolved
Hide resolved
findinpath
reviewed
Nov 14, 2022
...rino-product-tests/src/main/java/io/trino/tests/product/hudi/TestHudiSparkCompatibility.java
Outdated
Show resolved
Hide resolved
codope
force-pushed
the
hudi-product-tests
branch
from
4cebd70 to
8cece3a
Compare
Contributor
Author
|
@findinpath Addressed all your feedback. This PR is ready to land. |
ebyhr
approved these changes
Nov 15, 2022
...auncher/src/main/java/io/trino/tests/product/launcher/env/environment/EnvSinglenodeHudi.java
Outdated
Show resolved
Hide resolved
...n/resources/docker/presto-product-tests/conf/environment/singlenode-hudi/spark-defaults.conf
Outdated
Show resolved
Hide resolved
...rino-product-tests/src/main/java/io/trino/tests/product/hudi/TestHudiSparkCompatibility.java
Outdated
Show resolved
Hide resolved
...rino-product-tests/src/main/java/io/trino/tests/product/hudi/TestHudiSparkCompatibility.java
Outdated
Show resolved
Hide resolved
...rino-product-tests/src/main/java/io/trino/tests/product/hudi/TestHudiSparkCompatibility.java
Outdated
Show resolved
Hide resolved
...rino-product-tests/src/main/java/io/trino/tests/product/hudi/TestHudiSparkCompatibility.java
Outdated
Show resolved
Hide resolved
...rino-product-tests/src/main/java/io/trino/tests/product/hudi/TestHudiSparkCompatibility.java
Outdated
Show resolved
Hide resolved
...rino-product-tests/src/main/java/io/trino/tests/product/hudi/TestHudiSparkCompatibility.java
Outdated
Show resolved
Hide resolved
...rino-product-tests/src/main/java/io/trino/tests/product/hudi/TestHudiSparkCompatibility.java
Outdated
Show resolved
Hide resolved
...rino-product-tests/src/main/java/io/trino/tests/product/hudi/TestHudiSparkCompatibility.java
Outdated
Show resolved
Hide resolved
codope
force-pushed
the
hudi-product-tests
branch
from
9ce029b to
62a8798
Compare
ebyhr
approved these changes
Nov 17, 2022
...main/resources/docker/presto-product-tests/conf/environment/singlenode-hudi/log4j.properties
Outdated
Show resolved
Hide resolved
...rino-product-tests/src/main/java/io/trino/tests/product/hudi/TestHudiSparkCompatibility.java
Outdated
Show resolved
Hide resolved
...rino-product-tests/src/main/java/io/trino/tests/product/hudi/TestHudiSparkCompatibility.java
Outdated
Show resolved
Hide resolved
Member
|
Please squash commits into one. |
codope
force-pushed
the
hudi-product-tests
branch
from
62a8798 to
52cd4ec
Compare
codope
force-pushed
the
hudi-product-tests
branch
from
52cd4ec to
a6c25ae
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #14365
Description
Non-technical explanation
Release notes
(x) This is not user-visible or docs only and no release notes are required.
( ) Release notes are required, please propose a release note for me.
( ) Release notes are required, with the following suggested text: