Skip to content

Add concurrent writes reconciliation for OPTIMIZE in Delta Lake #22443

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 4 commits into from
Jan 14, 2025
Merged

Add concurrent writes reconciliation for OPTIMIZE in Delta Lake #22443

merged 4 commits into from
Jan 14, 2025

Conversation

@findinpath
Copy link
Contributor

@findinpath findinpath commented Jun 19, 2024 •
edited by raunaqmorarka
Loading

Description

Allow committing OPTIMIZE operations in a concurrent context
by placing these operations right after any other previously
concurrently completed write operations.

Additional context and related issues

Fixes #16985

Release notes

( ) This is not user-visible or is docs only, and no release notes are required.
( ) Release notes are required. Please propose a release note for me.
(x) Release notes are required, with the following suggested text:

# Delta Lake 
* Add support for concurrent execution of `OPTIMIZE`. ({issue}`16985`)

@cla-bot cla-bot bot added the cla-signed label Jun 19, 2024
@github-actions github-actions bot added the delta-lake Delta Lake connector label Jun 19, 2024
@findinpath findinpath mentioned this pull request Jun 19, 2024
Closed
@findinpath findinpath force-pushed the findinpath/delta-lake-optimize-reconciliation branch 2 times, most recently from c3fe9e8 to 9264397 Compare June 19, 2024 15:09
@findinpath findinpath marked this pull request as ready for review June 19, 2024 15:09
@findinpath findinpath force-pushed the findinpath/delta-lake-optimize-reconciliation branch 2 times, most recently from f4be087 to e695b50 Compare June 19, 2024 15:14
@findinpath findinpath requested review from ebyhr and pajaks June 19, 2024 15:14
pajaks
pajaks reviewed Jun 24, 2024
View reviewed changes
@github-actions
Copy link

This pull request has gone a while without any activity. Tagging the Trino developer relations team: @bitsondatadev @colebow @mosabua

@github-actions github-actions bot added the stale label Jul 15, 2024
@mosabua mosabua added stale-ignore Use this label on PRs that should be ignored by the stale bot so they are not flagged or closed. and removed stale labels Jul 15, 2024
@mosabua
Copy link
Member

mosabua commented Jul 15, 2024

Added stale-ignore label under the assumption that you will continue to work on this @findinpath

@findinpath findinpath force-pushed the findinpath/delta-lake-optimize-reconciliation branch 2 times, most recently from 20516f4 to c22ae88 Compare December 27, 2024 15:24
raunaqmorarka
raunaqmorarka approved these changes Jan 13, 2025
View reviewed changes
plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/DeltaLakeMetadata.java Outdated
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this commit change logic in any way, or just a refactoring ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just refactoring

@raunaqmorarka
Copy link
Member

/test-with-secrets sha=c22ae88bc6037a19e394e07aea155c3fdce147ed

@findinpath
Use only the enforced partition constraints dependency
f6447ad
@findinpath
Fix typo
6863f58
@findinpath
Extract logic for appending transaction log file for OPTIMIZE
c86cdb9
@findinpath
Add concurrent writes reconciliation for OPTIMIZE in Delta Lake
1158cb8 
Allow committing OPTIMIZE operations in a concurrent context
by placing these operations right after any other previously
concurrently completed write operations.
@findinpath findinpath force-pushed the findinpath/delta-lake-optimize-reconciliation branch from 22ff1a3 to 1158cb8 Compare January 13, 2025 16:00
@findinpath
Copy link
Contributor Author

Clean rebase on master only to avoid potentially introducing issues due to outdated code.

@raunaqmorarka
Copy link
Member

/test-with-secrets sha=1158cb8c04b7c1691292fefb630cbb7c91204628

@github-actions
Copy link

The CI workflow run with tests that require additional secrets has been started: https://github.com/trinodb/trino/actions/runs/12751874623

@raunaqmorarka raunaqmorarka merged commit f548262 into trinodb:master Jan 14, 2025
24 checks passed
@github-actions github-actions bot added this to the 469 milestone Jan 14, 2025
@mosabua mosabua mentioned this pull request Jan 14, 2025
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@raunaqmorarka raunaqmorarka raunaqmorarka approved these changes

@ebyhr ebyhr Awaiting requested review from ebyhr

@pajaks pajaks Awaiting requested review from pajaks

Assignees

No one assigned

Labels

cla-signed delta-lake Delta Lake connector stale-ignore Use this label on PRs that should be ignored by the stale bot so they are not flagged or closed.

Milestone

469

Development

Successfully merging this pull request may close these issues.

Concurrent writes reconciliation in Delta Lake

4 participants

@findinpath @mosabua @raunaqmorarka @pajaks