Skip to content

Draft: Add singlenode-hudi product test environment#14669

Closed
findinpath wants to merge 1 commit intotrinodb:masterfrom
findinpath:findinpath/spark3-hudi-product-tests
Closed

Draft: Add singlenode-hudi product test environment#14669
findinpath wants to merge 1 commit intotrinodb:masterfrom
findinpath:findinpath/spark3-hudi-product-tests

Conversation

@findinpath
Copy link
Copy Markdown
Contributor

@findinpath findinpath commented Oct 17, 2022 •
edited
Loading

Introduction

The current draft is based (same as for Delta Lake OSS) on a MinIO data lake because I think this kind of setup is close to the integration with AWS S3 which we want to test with Trino for Hudi.

Testing via Apache Hive JDBC Driver

Connect via Apache Hive driver to jdbc:hive2://localhost:10213/default
User hive , password na

show tables;

create table hudi_cow_nonpcf_tbl (
                                     uuid int,
                                     name string,
                                     price double
) using hudi
location 's3://presto-ci-test/hudi_cow_nonpcf_tbl';


select * from hudi_cow_nonpcf_tbl;

insert into hudi_cow_nonpcf_tbl select 1, 'a1', 20;

Testing via ptl

testing/bin/ptl test run --environment singlenode-hudi -- -t io.trino.tests.product.hudi.TestHudiCompatibility

@cla-bot cla-bot bot added the cla-signed label Oct 17, 2022
@findinpath findinpath force-pushed the findinpath/spark3-hudi-product-tests branch from 36f7226 to f23f182 Compare October 17, 2022 21:56
@findinpath findinpath mentioned this pull request Oct 17, 2022
Merged
@findinpath findinpath force-pushed the findinpath/spark3-hudi-product-tests branch from f23f182 to 00471b9 Compare October 17, 2022 22:08
findinpath
findinpath commented Oct 17, 2022
View reviewed changes
testing/trino-product-tests/src/main/java/io/trino/tests/product/utils/QueryExecutors.java Outdated
Copy link
Copy Markdown
Contributor Author

@findinpath findinpath Oct 17, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that a retry policy (as it can be seen for the other runners) is currently missing.
In case there will be identified cases worthy to apply a retry policy, then this call can be made with FailSafe.

@findinpath findinpath force-pushed the findinpath/spark3-hudi-product-tests branch 2 times, most recently from 3922aff to c757f93 Compare October 18, 2022 13:15
@findinpath findinpath force-pushed the findinpath/spark3-hudi-product-tests branch from c757f93 to 7884267 Compare November 3, 2022 06:44
ebyhr
ebyhr reviewed Nov 4, 2022
View reviewed changes
Copy link
Copy Markdown
Member

@ebyhr ebyhr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Docker dependency was upgraded to 72. Could you rebase on upstream?

...ing/trino-product-tests/src/main/java/io/trino/tests/product/hudi/TestHudiCompatibility.java
}

@Test(groups = {HUDI, PROFILE_SPECIFIC_TESTS})
public void testDemo()
Copy link
Copy Markdown
Member

@ebyhr ebyhr Nov 4, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

testDemo sounds little weird. I would rename to other name.

Copy link
Copy Markdown
Contributor Author

@findinpath findinpath Nov 4, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I intended with this PR (initially) only to test the hudi test docker image trinodb/docker-images#136

As I agreed (offline) with @codope, this PR can be used as base for building the actual hudi product tests.

...ing/trino-product-tests/src/main/java/io/trino/tests/product/hudi/TestHudiCompatibility.java
String tableName = "test_hudi_demo_" + randomTableSuffix();
String tableDirectory = "hudi-compatibility-test-" + tableName;

onHudi().executeQuery(format("CREATE TABLE default.%s (uuid int, col string) USING hudi LOCATION 's3://%s/%s'",
Copy link
Copy Markdown
Member

@ebyhr ebyhr Nov 4, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why uuid int? Other name looks better.

Copy link
Copy Markdown
Contributor Author

@findinpath findinpath Nov 4, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://hudi.apache.org/docs/quick-start-guide

uuid is used as primary key by default

...ing/trino-product-tests/src/main/java/io/trino/tests/product/hudi/TestHudiCompatibility.java
Comment on lines +56 to +57
onHudi().executeQuery("insert into default." + tableName + " select 1, 'Trino'");
onHudi().executeQuery("insert into default." + tableName + " select 2, 'rocks'");
Copy link
Copy Markdown
Member

@ebyhr ebyhr Nov 4, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Uppercase insert into & select.

@codope codope mentioned this pull request Nov 7, 2022
Merged
@ebyhr
Copy link
Copy Markdown
Member

ebyhr commented Nov 10, 2022

Superseded by #14936

@ebyhr ebyhr closed this Nov 10, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@electrum electrum electrum left review comments

@ebyhr ebyhr ebyhr left review comments

Assignees

No one assigned

Labels

cla-signed

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@findinpath @ebyhr @electrum