Test Delta Lake connector against Databricks runtime by findinpath · Pull Request #12896 · trinodb/trino

findinpath · 2022-06-18T05:57:51Z

Delta Lake Databricks Product Tests infrastructure

We currently have in Trino Delta Lake product tests for Delta OSS.
This PR aims to add support for Delta Lake product tests for Databricks environment as well.

The product test environment used to test Delta Lake connector functionality on top of Databricks relies on:

AWS S3: a S3 bucket which will hold the content of the delta tables
AWS Glue: metastore for the delta tables which can be accessed also by Trino (opposed to the internal Databricks default metastore)
Databricks

SuiteDeltaLake has been renamed to SuiteDeltaLakeOss and contains virtually no changes compared to master branch.

SuiteDeltaLakeDatabricks contains the Delta Lake tests to be executed on top of Databricks infrastructure.
This suite needs special credentials in order to perform operations on top of Databricks:

credentials for AWS
credentials for Databricks

A new job delta-lake-databricks-pt has been added to ci.yml.
This job is to be executed only when the credentials required by the SuiteDeltaLakeDatabricks are provided.

.github/workflows/ci.yml

findinpath · 2022-06-20T10:36:42Z

...a/io/trino/tests/product/launcher/env/environment/AbstractSinglenodeDeltaLakeDatabricks.java

This variable name can be a bit misleading.

I'd advocate for changing the name of the variable to : DATABRICKS104_TEST_JDBC_URL and also changing the class name EnvSinglenodeDeltaLakeDatabricks to EnvSinglenodeDeltaLakeDatabricks104

plugin/trino-delta-lake/README.md

...cker/presto-product-tests/conf/environment/singlenode-delta-lake-databricks/delta.properties

findepi · 2022-06-20T12:14:16Z

...tests/src/main/java/io/trino/tests/product/deltalake/TestDatabricksCompatibilityCleanUp.java

Rename constant to match new meaning. Separate commit.

findepi · 2022-06-20T12:14:59Z

...tests/src/main/java/io/trino/tests/product/deltalake/TestDatabricksCompatibilityCleanUp.java

before the change, what did this code execute against?

This code was never executed before in the context of Trino.
This code has been open sourced along with the other product tests, but not used until now.

...cker/presto-product-tests/conf/environment/singlenode-delta-lake-databricks/delta.properties

...tests/src/main/java/io/trino/tests/product/deltalake/TestDatabricksCompatibilityCleanUp.java

alexjo2144 · 2022-06-21T19:10:20Z

.github/workflows/ci.yml

If you take the TRINO_ prefix off aws clients will pick them up automatically

My intention was to avoid colliding with AWS credentials used by default for the CI runner for other product tests.

Do we have AWS credentials used by default for the CI?

We still may want to configure env for the containers only, and not the PTL, but it seems like a redundant complication to me.

I was under the impression that there are s3 / glue related product tests, but now after looking after any product test environments that have these needs, I didn't find any.
I can change the name of the variables, but they still need to be passed to the corresponding containers.

I was under the impression that there are s3 / glue related product tests,

They should exist, but they don't yet #5426

Once we have them, we will be invoked explicitly in ci.yml (my assumption)
and then they can set same env vars (with potentially different values)

we would not have ability to define one suite that exercises Hive+Glue and Delta+Databricks, but i don't know whether it will be a loss actually.

I can change the name of the variables, but they still need to be passed to the corresponding containers.

of course

plugin/trino-delta-lake/README.md

.github/workflows/ci.yml

...cker/presto-product-tests/conf/environment/singlenode-delta-lake-databricks/delta.properties

.github/workflows/ci.yml

plugin/trino-delta-lake/README.md

.github/workflows/ci.yml

findepi · 2022-06-24T10:21:35Z

Test PR with secrets: #12973
(cc @ilfrin @nineinchnick for potentially streamlining this)

nineinchnick · 2022-06-24T10:23:23Z

Test PR with secrets: #12973 (cc @ilfrin @nineinchnick for potentially streamlining this)

#12817 is waiting for your review

findinpath · 2022-06-27T04:48:24Z

Rebased on master to include the latest Delta Lake product tests in the build.

As part of the open sourcing process of the product tests for the `trino-delta-lake` connector, the product tests dedicated for the Databricks runtime were dependent on Hive Metastore Service. However, using AWS Glue comes with the advantage of not managing an extra service for running product tests against the Databricks runtime. This is why the product tests are now adapted to run with AWS Glue as metastore service for the Databricks runtime.

When trying to create tables on Databricks on the schema `extraordinary`, the creation of the tables was failing with the message: ``` Error in SQL statement: IllegalArgumentException: Path must be absolute: test100-__PLACEHOLDER__ ``` Specifying the location for the custom schema helps avoiding the above-mentioned issue.

The tests in the class TestHiveAndDeltaLakeRedirect run on both Delta OSS and Databricks infrastructure and not only on Databricks.

Avoid running into a test failure when dealing with a test depending on Databricks AWS S3 and AWS Glue environment. Consult for further details: trinodb#13017

Tables created for testing purposes on AWS Glue / AWS S3 have to be eventually removed. Use the prefix "test_" to identify easier the tests in the routine responsible for removing the test Glue tables.

ebyhr · 2022-07-04T11:54:52Z

Merged, thanks!

cla-bot bot added the cla-signed label Jun 18, 2022

findinpath marked this pull request as draft June 18, 2022 05:58

github-actions bot added the tests:hive label Jun 18, 2022

findinpath force-pushed the delta-product-tests-databricks branch from 1305140 to 5b35f90 Compare June 20, 2022 08:31

findinpath requested review from alexjo2144, ebyhr, findepi, hashhar and nineinchnick June 20, 2022 09:15

findinpath marked this pull request as ready for review June 20, 2022 09:23

nineinchnick reviewed Jun 20, 2022

View reviewed changes

.github/workflows/ci.yml Outdated Show resolved Hide resolved

findinpath commented Jun 20, 2022

View reviewed changes

findepi reviewed Jun 20, 2022

View reviewed changes

findinpath force-pushed the delta-product-tests-databricks branch 3 times, most recently from 59a2adc to 7c9b827 Compare June 21, 2022 09:06

alexjo2144 reviewed Jun 21, 2022

View reviewed changes

findinpath force-pushed the delta-product-tests-databricks branch from 7c9b827 to 141baa3 Compare June 22, 2022 04:47

findepi reviewed Jun 22, 2022

View reviewed changes

findinpath force-pushed the delta-product-tests-databricks branch from 141baa3 to 631bd69 Compare June 23, 2022 07:11

findepi reviewed Jun 23, 2022

View reviewed changes

.github/workflows/ci.yml Outdated Show resolved Hide resolved

findinpath force-pushed the delta-product-tests-databricks branch from 631bd69 to 1c2bb20 Compare June 24, 2022 03:25

findepi mentioned this pull request Jun 24, 2022

Test: Adapt Delta product tests for execution on top of Databricks backed by AWS Glue #12973

Closed

findinpath force-pushed the delta-product-tests-databricks branch 2 times, most recently from fd783e3 to 3291132 Compare June 27, 2022 04:47

findinpath force-pushed the delta-product-tests-databricks branch from de441ae to 6a07350 Compare June 28, 2022 15:22

Remove TEST token from the name of Databricks variables

d4393c3

Adapt EnvSinglenodeDeltaLakeDatabricks to Databricks 7.3 environment

4fbcb5c

findinpath force-pushed the delta-product-tests-databricks branch 2 times, most recently from 63d5852 to 6f3425b Compare June 29, 2022 16:14

findinpath added 6 commits July 1, 2022 12:41

Drop assertions for the UPDATE on Spark

53a13a0

Rename method createTableInDatabricks to createTableOnDelta

76326fd

The tests in the class TestHiveAndDeltaLakeRedirect run on both Delta OSS and Databricks infrastructure and not only on Databricks.

Exclude test from TestDeltaLakeDropTableCompatibility class

93c440d

Avoid running into a test failure when dealing with a test depending on Databricks AWS S3 and AWS Glue environment. Consult for further details: trinodb#13017

Handle eventual failures in cleaning up after table rename

213470f

findinpath force-pushed the delta-product-tests-databricks branch from 6f3425b to d63456f Compare July 1, 2022 10:46

findinpath changed the title ~~Adapt Delta product tests for execution on top of Databricks backed by AWS Glue~~ Test Delta Lake connector against Databricks runtime Jul 1, 2022

findinpath force-pushed the delta-product-tests-databricks branch from d63456f to 4b428f9 Compare July 1, 2022 13:45

Prefix table names to underline their testing nature

4fbf790

Tables created for testing purposes on AWS Glue / AWS S3 have to be eventually removed. Use the prefix "test_" to identify easier the tests in the routine responsible for removing the test Glue tables.

findinpath force-pushed the delta-product-tests-databricks branch from 4b428f9 to e73186f Compare July 1, 2022 13:59

Test Delta Lake connector against Databricks runtime

bd49f3c

findinpath force-pushed the delta-product-tests-databricks branch from e73186f to bd49f3c Compare July 4, 2022 09:44

ebyhr approved these changes Jul 4, 2022

View reviewed changes

ebyhr merged commit e4093da into trinodb:master Jul 4, 2022

github-actions bot added this to the 389 milestone Jul 4, 2022

colebow mentioned this pull request Jul 5, 2022

Add Trino 389 release notes #13089

Merged

ebyhr mentioned this pull request Jul 8, 2022

Add Delta Lake product tests #11369

Closed

Conversation

findinpath commented Jun 18, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Delta Lake Databricks Product Tests infrastructure

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

findepi commented Jun 24, 2022

Uh oh!

nineinchnick commented Jun 24, 2022

Uh oh!

findinpath commented Jun 27, 2022

Uh oh!

ebyhr commented Jul 4, 2022

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

5 participants

findinpath commented Jun 18, 2022 •

edited

Loading