Skip to content

Filter Delta Lake splits based on $path column predicates#13172

Merged
ebyhr merged 1 commit intotrinodb:masterfrom
ebyhr:ebi/delta-path-predicate
Jul 14, 2022
Merged

Filter Delta Lake splits based on $path column predicates#13172
ebyhr merged 1 commit intotrinodb:masterfrom
ebyhr:ebi/delta-path-predicate

Conversation

@ebyhr
Copy link
Copy Markdown
Member

@ebyhr ebyhr commented Jul 14, 2022

Description

Filter Delta Lake splits based on $path column predicates
Fixes #13169

Documentation

(x) No documentation is needed.

Release notes

(x) Release notes entries required with the following suggested text:

# Delta Lake
* Support filtering splits based on `$path` column predicates. ({issue}`13169`)

@cla-bot cla-bot bot added the cla-signed label Jul 14, 2022
@ebyhr ebyhr force-pushed the ebi/delta-path-predicate branch from 15fda95 to 8c7740c Compare July 14, 2022 06:25
@ebyhr ebyhr merged commit bc7a4ee into trinodb:master Jul 14, 2022
@ebyhr ebyhr deleted the ebi/delta-path-predicate branch July 14, 2022 07:38
@ebyhr ebyhr mentioned this pull request Jul 14, 2022
@github-actions github-actions bot added this to the 390 milestone Jul 14, 2022
@alexjo2144
Copy link
Copy Markdown
Member

We could also make a change in DeltaLakeMetadata#applyFilter to put $path predicates in the enforcedConstraint

@ebyhr
Copy link
Copy Markdown
Member Author

ebyhr commented Jul 15, 2022

I didn't touch the method intentionally because it's enforced"Partition"Constraint in DeltaLakeTableHandle. We can improve it later.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Development

Successfully merging this pull request may close these issues.

Filter Delta Lake splits based on metadata column predicates

3 participants