Add support for writing partitionValues_parsed for the add entries in the Delta Lake checkpoint#19662
Merged
ebyhr merged 4 commits intotrinodb:masterfrom Nov 30, 2023
Conversation
findinpath
force-pushed
the
findinpath/delta-write-part-values-parsed
branch
from
00ab843 to
d17b1f7
Compare
findinpath
force-pushed
the
findinpath/delta-write-part-values-parsed
branch
2 times, most recently
from
94fb6e1 to
d49d727
Compare
Contributor
Author
|
@ebyhr pls do run the PR with secrets. |
findinpath
changed the title
partitionValues_parsed in the Delta Lake checkpointpartitionValues_parsed for the add entries in the Delta Lake checkpoint
Member
|
/test-with-secrets sha=d49d727bc2d0a1aac8968aaa6a9bbb7af724781d |
|
The CI workflow run with tests that require additional secrets finished as failure: https://github.com/trinodb/trino/actions/runs/6818424653 |
findinpath
force-pushed
the
findinpath/delta-write-part-values-parsed
branch
3 times, most recently
from
e54e63a to
406aaf3
Compare
Member
|
Could you rebase on master to resolve conflicts? |
findinpath
force-pushed
the
findinpath/delta-write-part-values-parsed
branch
from
406aaf3 to
7df2c68
Compare
raunaqmorarka
approved these changes
Nov 27, 2023
...lake/src/main/java/io/trino/plugin/deltalake/transactionlog/checkpoint/CheckpointWriter.java
Outdated
Show resolved
Hide resolved
plugin/trino-delta-lake/src/test/java/io/trino/plugin/deltalake/TestDeltaLakeConnectorTest.java
Outdated
Show resolved
Hide resolved
...ts/src/main/java/io/trino/tests/product/deltalake/TestDeltaLakeCheckpointsCompatibility.java
Outdated
Show resolved
Hide resolved
findinpath
force-pushed
the
findinpath/delta-write-part-values-parsed
branch
from
7df2c68 to
f5d9db2
Compare
findinpath
force-pushed
the
findinpath/delta-write-part-values-parsed
branch
from
f5d9db2 to
14e9de8
Compare
Contributor
Author
|
CI hit #16315 |
ebyhr
reviewed
Nov 29, 2023
Member
There was a problem hiding this comment.
Is TIMESTAMP_NTZ type disallowed as a partition column? It would be nice to create the table in Spark and cover the type if it's supported.
Contributor
Author
There was a problem hiding this comment.
Added corresponding test in TestDeltaLakeBasic.
Member
There was a problem hiding this comment.
Thanks, but TestDeltaLakeBasic doesn't ensure that the field in checkpoint is readable by Spark.
...ts/src/main/java/io/trino/tests/product/deltalake/TestDeltaLakeCheckpointsCompatibility.java
Outdated
Show resolved
Hide resolved
...ts/src/main/java/io/trino/tests/product/deltalake/TestDeltaLakeCheckpointsCompatibility.java
Outdated
Show resolved
Hide resolved
...ts/src/main/java/io/trino/tests/product/deltalake/TestDeltaLakeCheckpointsCompatibility.java
Outdated
Show resolved
Hide resolved
...lake/src/main/java/io/trino/plugin/deltalake/transactionlog/checkpoint/CheckpointWriter.java
Outdated
Show resolved
Hide resolved
findinpath
force-pushed
the
findinpath/delta-write-part-values-parsed
branch
2 times, most recently
from
6885610 to
60b723b
Compare
findinpath
force-pushed
the
findinpath/delta-write-part-values-parsed
branch
from
60b723b to
d585417
Compare
Contributor
Author
|
@ebyhr AC |
ebyhr
approved these changes
Nov 30, 2023
Member
There was a problem hiding this comment.
Thanks, but TestDeltaLakeBasic doesn't ensure that the field in checkpoint is readable by Spark.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Write
partitionValues_parsedfield for theaddentries in the Delta Lake checkpoint file.This information can be used to massively reduce the amount of information read from the checkpoint file in case the
SELECTquery from the Delta Lake table has a partition filter. See #19588 for details.Fixes #19586
Additional context and related issues
Release notes
(x) This is not user-visible or is docs only, and no release notes are required.