Skip to content

Conversation

@bartash
Copy link
Member

@bartash bartash commented Dec 5, 2023

This is a clean cherry-pick of #9182 to 1.4.x
Closes #9182
cc @Fokko

…che#9183)

When a snapshot is expired as part of a transaction, the manifest list
file(s) should be deleted when the transaction commits. A recent change
(apache#6634) ensured that these files are not deleted when they have also
been committed as part of a transaction, but this breaks the simple
case where no new files are committed. Fix this by not skipping
deletion when the list of committed files is empty.

TESTING:

Extended a unit test to ensure that manifest list files are deleted.
Ran the test without the fix on a branch where apache#6634 was reverted
to show that this is a regression.
@amogh-jahagirdar
Copy link
Contributor

Thanks @bartash . I think let's get #9221 in first and then we can rebase this change? The change for returning a new empty set was not just stylistic, it was intentional to distinguish the case where deletion can be performed (empty snapshot list from expire snapshots in a transaction) and where deletion should not be performed if we could not determine the files to delete (snapshot lookup failed)

@bartash
Copy link
Member Author

bartash commented Dec 5, 2023

Thanks @amogh-jahagirdar sounds good

Copy link
Contributor

@amogh-jahagirdar amogh-jahagirdar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good to me. Thanks for taking this @bartash appreciate it!

@amogh-jahagirdar amogh-jahagirdar added this to the Iceberg 1.4.3 milestone Dec 6, 2023
@nastra nastra changed the title Core: Expired Snapshot files in a transaction should be deleted (cherry-pick to 1.4.x) [1.4.x] Core: Expired Snapshot files in a transaction should be deleted Dec 19, 2023
@nastra nastra requested a review from Fokko December 19, 2023 07:57
@amogh-jahagirdar amogh-jahagirdar merged commit 63e0204 into apache:1.4.x Dec 19, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants