Skip to content

Do not run Hive and Delta compatibility tests on Databricks#18878

Merged
ebyhr merged 1 commit intotrinodb:masterfrom
findinpath:findinpath/delta-hive-redirect-pt
Aug 31, 2023
Merged

Do not run Hive and Delta compatibility tests on Databricks#18878
ebyhr merged 1 commit intotrinodb:masterfrom
findinpath:findinpath/delta-hive-redirect-pt

Conversation

@findinpath
Copy link
Copy Markdown
Contributor

@findinpath findinpath commented Aug 31, 2023

Description

Test against Delta Lake OSS gives enough certainty for the accuracy
of the Delta Lake connector.

Relates to #18805

Additional context and related issues

Release notes

(x) This is not user-visible or is docs only, and no release notes are required.

@cla-bot cla-bot bot added the cla-signed label Aug 31, 2023
@findinpath findinpath requested review from ebyhr and pajaks August 31, 2023 07:22
@findinpath findinpath added the no-release-notes This pull request does not require release notes entry label Aug 31, 2023
Copy link
Copy Markdown
Member

@ebyhr ebyhr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we still need hive.properties in AbstractSinglenodeDeltaLakeDatabricks & singlenode-delta-lake-databricks directory and also delta.hive-catalog-name=hive in delta.properties?

@findinpath findinpath changed the title Do not run Hive and Delta redirect tests on Databricks Do not run Hive and Delta compatibility tests on Databricks Aug 31, 2023
@findinpath
Copy link
Copy Markdown
Contributor Author

findinpath commented Aug 31, 2023

Do we still need hive.properties in AbstractSinglenodeDeltaLakeDatabricks & singlenode-delta-lake-databricks directory and also delta.hive-catalog-name=hive in delta.properties?

I'm trying to figure out whether we need

onTrino().executeQuery("SET SESSION hive.hive_views_legacy_translation = true");

Let's handle this in a follow-up PR to focus solely on this aspect.
In any case TestDatabricksWithGlueMetastoreCleanUp should be probably executed only once per build, not for every databricks product test.

Idea for smoke test to legitimate existence of hive.properties: We could have a hive and delta lake compatibility test where we can observe whether Hive tables (USING parquet) created via Databricks are readable through the hive connector.

Test against Delta Lake OSS gives enough certainty for the accuracy
of the Delta Lake connector.
@findinpath findinpath force-pushed the findinpath/delta-hive-redirect-pt branch from b642f33 to c999c2f Compare August 31, 2023 07:39
@ebyhr ebyhr merged commit c23c8f7 into trinodb:master Aug 31, 2023
@github-actions github-actions bot added this to the 426 milestone Aug 31, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cla-signed no-release-notes This pull request does not require release notes entry

Development

Successfully merging this pull request may close these issues.

2 participants