Core: Fix ancestor lookup during expire file cleanup #5666
Conversation
…se the ancestors of the only existing branch Until reachability analysis for remove snapshots is implemented
| // only remove files that were deleted in an ancestor of the current table state to avoid | ||
| // ToDo: This will be removed when reachability analysis is done so files across multiple | ||
| // branches can be removed | ||
| SnapshotRef branchToCleanup = Iterables.getFirst(base.refs().values(), null); |
There was a problem hiding this comment.
Here the expectation is that only one ref exists. Either main or branch ref. What if it's a tag ref?
There was a problem hiding this comment.
My thinking is the following:
1.) Logically, a tagged snapshot would either need to exist on either a.) non-main branch b.) main-branch
2.) If the tag exists on main a file cleanup couldn't (currently) be done in the first place (because main cannot age off so we'd have multiple refs), so this point wouldn't have been reached
3.) If the tag exists on a non-main branch and the non-main branch ages off before the tagged snapshot which gets retained, then the tag ends up being de-facto "tip" of a lineage. In which case, the expiration logic would work as expected. If non-main branch still is retained, then we wouldn't reach this point (same case as 2, just that the other ref is the non-main branch).
Combining this with the fact that writes cannot be performed on tags leads me to believe that for purpose of expiration , specifically determining which files to delete, there's no need to differentiate tags and branches.
I could call this refToCleanup if that makes more sense to folks? But the only case where this is a tag is the case what I mentioned in 3.) in which case it's just a "dangling" snapshot which is referenced by a tag. @namrathamyske @rdblue @jackye1995
Also let me know if there's a flaw in my logic
rdblue
changed the title
|
|
||
| // this is the set of ancestors of the current table state. when removing snapshots, this must | ||
| // only remove files that were deleted in an ancestor of the current table state to avoid | ||
| // ToDo: This will be removed when reachability analysis is done so files across multiple |
3 participants
Currently, clean up of files can only occur in ExpireSnapshots if there's 1 reference. It can be either main or a single non-main branch.
However, the ancestor lookup that's done is done based on the main table state, so this will lead to unexpected file deletions when the only branch is a non-main branch and delete files are performed on the branch. The PR for delete files is here but I think this fix is universal until a reachability analysis is done.
When updating expire snapshot tests which test branch deletions on a branch in this PR I encountered test failures due to data files being deleted which should not be for the non-main branch case. The snapshots which were getting expired were the expected snapshots, but the data files being deleted for the branch commit were unexpected because some of the manifests being reverted were unexpected because the check here would unexpectedly pass because the isFromAncestor would evaluate to false (and the rest of the checks were as expected), so the procedure would add manifests which should not be reverted to the reverted set.
CC: @rdblue @jackye1995 @namrathamyske