Core: pushdown data_file.content when filter manifests in entries table #10203

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Merged

szehon-ho merged 8 commits into apache:main from dramaticlly:pushDownEntries

Jun 24, 2024

Contributor

dramaticlly commented Apr 22, 2024 •

edited

Loading

Skip either data or delete manifest when reading entries table by filter on its data_file.content. Build a new ManifestContentEvaluator in entries metadata table.

Can you take a look ? @szehon-ho @aokolnychyi


          Core: pushdown data_file.content when filter manifests in entries table

26f0ccf

github-actions bot added the core label

szehon-ho reviewed

View reviewed changes

core/src/main/java/org/apache/iceberg/BaseEntriesTable.java Outdated Show resolved Hide resolved

core/src/main/java/org/apache/iceberg/BaseEntriesTable.java Outdated Show resolved Hide resolved

core/src/main/java/org/apache/iceberg/BaseEntriesTable.java Outdated Show resolved Hide resolved

core/src/main/java/org/apache/iceberg/BaseEntriesTable.java Outdated Show resolved Hide resolved


          Simplify the pushdown logic based on Szehon review feedback

65ea46b

szehon-ho reviewed

View reviewed changes

core/src/main/java/org/apache/iceberg/BaseEntriesTable.java Show resolved Hide resolved

szehon-ho reviewed

View reviewed changes

core/src/main/java/org/apache/iceberg/BaseEntriesTable.java Outdated Show resolved Hide resolved


          Add back notEq

b22b6a5

szehon-ho reviewed

View reviewed changes

core/src/main/java/org/apache/iceberg/BaseEntriesTable.java Outdated Show resolved Hide resolved


          Extract common logic

b56a91b

szehon-ho reviewed

View reviewed changes

core/src/main/java/org/apache/iceberg/BaseEntriesTable.java Show resolved Hide resolved

core/src/main/java/org/apache/iceberg/BaseEntriesTable.java Outdated Show resolved Hide resolved

core/src/test/java/org/apache/iceberg/TestMetadataTableScans.java Outdated Show resolved Hide resolved

core/src/test/java/org/apache/iceberg/MetadataTableScanTestBase.java Outdated Show resolved Hide resolved

dramaticlly added 2 commits

June 19, 2024 10:14


          Improve pushdown logic for both In and NotIn

a4cea44


          Improve pushdown logic for both In and NotIn

5a38422

szehon-ho reviewed

View reviewed changes

core/src/main/java/org/apache/iceberg/BaseEntriesTable.java Outdated Show resolved Hide resolved


          convert for loop to anyMatch

f51478c

szehon-ho reviewed

View reviewed changes

core/src/main/java/org/apache/iceberg/BaseEntriesTable.java Show resolved Hide resolved

core/src/main/java/org/apache/iceberg/BaseEntriesTable.java Outdated Show resolved Hide resolved

szehon-ho reviewed

View reviewed changes

core/src/main/java/org/apache/iceberg/BaseEntriesTable.java Outdated Show resolved Hide resolved

szehon-ho reviewed

View reviewed changes

core/src/main/java/org/apache/iceberg/BaseEntriesTable.java Show resolved Hide resolved

szehon-ho reviewed

View reviewed changes

core/src/test/java/org/apache/iceberg/MetadataTableScanTestBase.java Outdated Show resolved Hide resolved


          Address review feedback and simplify logic in in and notIn pushdown

43bc7e5

dramaticlly closed this

dramaticlly reopened this

dramaticlly closed this

dramaticlly reopened this

dramaticlly mentioned this pull request

Core: Add data sequence number as derived column to files metadata table #9813

Closed

szehon-ho approved these changes

View reviewed changes

szehon-ho merged commit 5733aec into apache:main

Member

szehon-ho commented Jun 24, 2024

Merged, thanks @dramaticlly

dramaticlly mentioned this pull request

Core, Spark: Remove dangling deletes as part of RewriteDataFilesAction #9724

Merged

jasonf20 pushed a commit to jasonf20/iceberg that referenced this pull request


          Core: Pushdown data_file.content filter in entries metadata table (ap…

c104187

…ache#10203)

dramaticlly deleted the pushDownEntries branch

December 12, 2024 19:05

zachdisc pushed a commit to zachdisc/iceberg that referenced this pull request


          Core: Pushdown data_file.content filter in entries metadata table (ap…

24404c9

…ache#10203)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core