Enclose CombinedDeletionPolicy in SnapshotDeletionPolicy #27528
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Today we can not distinguish between index commits that are kept by the primary policy and those are kept for snapshotting with a SnapshotDeletionPolicy. Since we enclose a SnapshotDeletionPolicy in a CombinedDeletionPolicy, we also we can not distinguish between those with a CombinedDeletionPolicy. This can be a problem if we update the TranslogDeletionPolicy to keep the minimum translog generation of undeleted index commits as we may keep the translog of a snapshotting commit even though it is "deleted" by the primary policy. To solve this, we enclose a CombinedDeletionPolicy in a SnapshotDeletionPolicy and track if an index commit is deleted by the primary policy, then use that value to maintain translog rather than the actual deletion of an index commit. Relates elastic#27456 elastic#27367
bleskes
reviewed
Nov 27, 2017
| * Wraps a provided {@link IndexCommit} and tracks if it has been deleted by the policy. | ||
| * This tracks if {@link #delete()} has been called or not even the actual action can be ignored by {@link SnapshotDeletionPolicy}. | ||
| */ | ||
| private static class TrackedIndexCommit extends IndexCommit { |
Contributor
There was a problem hiding this comment.
This sucks.. I want to see how big of a deal it is to keep things as they were. Indeed snapshotting a commit will keep it's translog around but I'm not sure anymore it's worth this kind of wrapping layers. Maybe we should invest in faster clean up on those snapshotted commits. I'll reach out to discuss.
Member
Author
I am closing this and will integrate this policy into the |
clintongormley
added
:Distributed Indexing/Distributed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Today we can not distinguish between index commits that are kept by the
primary policy and those are kept for snapshotting with a
SnapshotDeletionPolicy. Since we enclose a SnapshotDeletionPolicy in a
CombinedDeletionPolicy, we also we can not distinguish between those
with a CombinedDeletionPolicy. This can be a problem if we update the
TranslogDeletionPolicy to keep the minimum translog generation of
undeleted index commits as we may keep the translog of a snapshotting
commit even though it is "deleted" by the primary policy.
To solve this, we enclose a CombinedDeletionPolicy in a
SnapshotDeletionPolicy and track if an index commit is deleted by the
primary policy, then use that value to maintain translog rather than the
actual deletion of an index commit.
Relates #27456 #27367