Add support for writer version 7 in Delta Lake by ebyhr · Pull Request #18423 · trinodb/trino

ebyhr · 2023-07-26T06:33:17Z

Description

Release notes

(x) Release notes are required, with the following suggested text:

# Delta Lake
* Add support for writer version 7. ({issue}`15873`)

findinpath · 2023-08-22T15:32:36Z

Build is 🔴

Error:  Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.11.0:compile (default-compile) on project trino-product-tests: Compilation failure
Error:  /home/runner/work/trino/trino/testing/trino-product-tests/src/main/java/io/trino/tests/product/deltalake/TestDeltaLakeColumnMappingMode.java:[131,80] cannot find symbol
Error:    symbol:   variable ENGLISH
Error:    location: class io.trino.tests.product.deltalake.TestDeltaLakeColumnMappingMode

...rc/main/java/io/trino/tests/product/deltalake/TestDeltaLakeCheckConstraintCompatibility.java

ebyhr · 2023-09-01T05:26:39Z

Addressed comments.

ebyhr · 2023-09-04T09:36:38Z

Rebased on master to resolve conflicts.

plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/DeltaLakeMetadata.java

plugin/trino-delta-lake/src/test/resources/databricks/invariants_writer_feature/README.md

...rc/main/java/io/trino/tests/product/deltalake/TestDeltaLakeCheckConstraintCompatibility.java

...uct-tests/src/main/java/io/trino/tests/product/deltalake/TestDeltaLakeColumnMappingMode.java

...t-tests/src/main/java/io/trino/tests/product/deltalake/TestDeltaLakeDeleteCompatibility.java

...rc/main/java/io/trino/tests/product/deltalake/TestDeltaLakeWriteDatabricksCompatibility.java

...n/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/transactionlog/TableSnapshot.java

...lake/src/test/java/io/trino/plugin/deltalake/TestDeltaLakeMinioAndHmsConnectorSmokeTest.java

ebyhr · 2023-09-04T22:37:25Z

Addressed comments.

ebyhr · 2023-09-05T22:55:08Z

@findepi @alexjo2144 Could you take another look when you have time?

findepi · 2023-09-06T10:19:32Z

plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/DeltaLakeMetadata.java

            return null;
        }
-        verifySupportedColumnMapping(getColumnMappingMode(metadataEntry));
+        verifySupportedColumnMapping(getColumnMappingMode(metadataEntry, protocolEntry));


Can this change go as a separate prep commit?

Even if we separate a commit, the preparatory commit will contain almost all changes except for tests.

plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/DeltaLakePageSinkProvider.java

ebyhr · 2023-09-13T03:25:52Z

Rebased on master to resolve conflicts and added another commit to handle deletion vectors feature in protocol entry.

Support the following writer features: * appendOnly * invariants * checkConstraints * changeDataFeed * columnMapping

jkylling · 2023-09-22T15:28:58Z

...n/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/transactionlog/TableSnapshot.java

+            Iterator<DeltaLakeTransactionLogEntry> entries = getCheckpointTransactionLogEntries(
                    session,
-                    ImmutableSet.of(METADATA),
+                    ImmutableSet.of(METADATA, PROTOCOL),


This change likely breaks the checkpoint reading optimizations added in 06e1339

We should probably adopt the optimization to support or-ing multiple tuple domains.

Please file an issue to https://github.com/trinodb/trino/issues/new

cla-bot bot added the cla-signed label Jul 26, 2023

ebyhr self-assigned this Jul 26, 2023

github-actions bot added tests:hive delta-lake Delta Lake connector labels Jul 26, 2023

ebyhr force-pushed the ebi/delta-writer-7 branch 5 times, most recently from 8e63e0b to c6c4cad Compare July 31, 2023 01:17

ebyhr force-pushed the ebi/delta-writer-7 branch 4 times, most recently from 517a340 to 0557cbe Compare August 15, 2023 07:55

ebyhr marked this pull request as ready for review August 15, 2023 10:05

ebyhr force-pushed the ebi/delta-writer-7 branch 2 times, most recently from aa8b88d to 47e20fc Compare August 15, 2023 23:17

ebyhr requested review from findepi, findinpath and homar August 15, 2023 23:17

ebyhr force-pushed the ebi/delta-writer-7 branch from 47e20fc to 8585ab9 Compare August 22, 2023 12:44

ebyhr force-pushed the ebi/delta-writer-7 branch from 8585ab9 to 525d238 Compare August 22, 2023 21:39

ebyhr marked this pull request as draft August 22, 2023 23:05

ebyhr force-pushed the ebi/delta-writer-7 branch 5 times, most recently from 1b11c0a to 70b9faa Compare August 25, 2023 04:01

ebyhr marked this pull request as ready for review August 25, 2023 06:01

findinpath reviewed Aug 28, 2023

View reviewed changes

...rc/main/java/io/trino/tests/product/deltalake/TestDeltaLakeCheckConstraintCompatibility.java Show resolved Hide resolved

ebyhr mentioned this pull request Sep 4, 2023

Simplify assertion of SHOW CREATE TABLE result in Delta Lake tests #18904

Merged

ebyhr force-pushed the ebi/delta-writer-7 branch from fef0872 to 4f28fed Compare September 4, 2023 09:36

findinpath reviewed Sep 4, 2023

View reviewed changes

ebyhr mentioned this pull request Sep 5, 2023

Add support for writing timestamp_ntz type in Delta #18913

Merged

ebyhr force-pushed the ebi/delta-writer-7 branch from a40eb59 to 47b3bb9 Compare September 5, 2023 00:23

ebyhr mentioned this pull request Sep 5, 2023

Add ProtocolEntry to DeltaLakeTableHandle #18914

Merged

ebyhr force-pushed the ebi/delta-writer-7 branch from 47b3bb9 to fc24a16 Compare September 5, 2023 02:39

findinpath approved these changes Sep 5, 2023

View reviewed changes

ebyhr force-pushed the ebi/delta-writer-7 branch 2 times, most recently from 26f18f4 to 0b81e20 Compare September 5, 2023 11:19

ebyhr mentioned this pull request Sep 6, 2023

Simplify DeltaLakeMetadata with new model class #18133

Merged

findepi reviewed Sep 6, 2023

View reviewed changes

findinpath reviewed Sep 12, 2023

View reviewed changes

plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/DeltaLakePageSinkProvider.java Outdated Show resolved Hide resolved

findinpath approved these changes Sep 12, 2023

View reviewed changes

ebyhr force-pushed the ebi/delta-writer-7 branch from 0b81e20 to ee96b79 Compare September 13, 2023 03:25

findepi approved these changes Sep 13, 2023

View reviewed changes

Add support for writer version 7 in Delta Lake

dd46758

Support the following writer features: * appendOnly * invariants * checkConstraints * changeDataFeed * columnMapping

ebyhr force-pushed the ebi/delta-writer-7 branch from ee96b79 to dd46758 Compare September 14, 2023 08:55

ebyhr merged commit 6f15559 into master Sep 14, 2023

ebyhr deleted the ebi/delta-writer-7 branch September 14, 2023 22:02

github-actions bot added this to the 427 milestone Sep 14, 2023

jkylling reviewed Sep 22, 2023

View reviewed changes

colebow mentioned this pull request Sep 25, 2023

Add Trino 427 release notes #19023

Merged

ebyhr mentioned this pull request Sep 25, 2023

Restore predicate pushdown of metadata field in CheckpointEntryIterator #19157

Closed

findinpath mentioned this pull request Nov 6, 2023

Remove outdated TODO comment #19642

Merged

Conversation

ebyhr commented Jul 26, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Release notes

Uh oh!

findinpath commented Aug 22, 2023

Uh oh!

Uh oh!

ebyhr commented Sep 1, 2023

Uh oh!

ebyhr commented Sep 4, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ebyhr commented Sep 4, 2023

Uh oh!

ebyhr commented Sep 5, 2023

Uh oh!

findepi Sep 6, 2023

Choose a reason for hiding this comment

Uh oh!

ebyhr Sep 6, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ebyhr commented Sep 13, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jkylling Sep 22, 2023

Choose a reason for hiding this comment

Uh oh!

ebyhr Sep 25, 2023

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

5 participants

ebyhr commented Jul 26, 2023 •

edited

Loading

ebyhr commented Sep 13, 2023 •

edited

Loading