HDDS-10524. [Snapshot] Invalidate the cache entry from snapshotInfoTable cache in OMSnapshotPurgeRequest #6443
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
hemantk-12
changed the title
swamirishi
requested changes
Mar 26, 2024
Contributor
swamirishi
left a comment
swamirishi
left a comment
There was a problem hiding this comment.
Thanks for figuring out the issue and raising a patch for the same @hemantk-12
...ager/src/main/java/org/apache/hadoop/ozone/om/response/snapshot/OMSnapshotPurgeResponse.java
Outdated
Show resolved
Hide resolved
Contributor
Author
|
Thanks @swamirishi for the review. |
myskov
pushed a commit
to myskov/ozone
that referenced
this pull request
Apr 4, 2024
…ble cache in OMSnapshotPurgeRequest (apache#6443) (cherry picked from commit 7559e1f)
jojochuang
pushed a commit
to jojochuang/ozone
that referenced
this pull request
May 29, 2024
…ble cache in OMSnapshotPurgeRequest (apache#6443) (cherry picked from commit 7559e1f)
swamirishi
pushed a commit
to swamirishi/ozone
that referenced
this pull request
Jun 10, 2024
…apshotInfoTable cache in OMSnapshotPurgeRequest (apache#6443) Change-Id: I96831f1f18f092950a2fbf257571d7a587e13f8a
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Currently when snapshot is purged, it is not removed from the snapshotInfoTable cache. Because of which it is possible that a purged snapshot is added by
SnapshotSetPropertyAPI.KeyDeletingServiceandSnapshotDeletingServiceare two background service which run in parallel.1). KeyDeletingService updates deep clean flag and exclusive size for a snapshot.
2). SnapshotDeletingService purges a deleted snapshot.
KeyDeletionService will try to mark a snapshot deep clean here: code
Since it is in a loop, it is possible when KeyDeletingService is going over all the snapshot. Meanwhile user deletes the snapshot, SnapshotDeletingService kicks in and purge the snapshot code here.
Now when
OMSnapshotSetPropertyRequestgets request to update deep clean for the same snapshot (purged by SnapshotDeletingService), it gets the value from cache because we don’t purge snapshot from cache on the purge path and wait for double buffer flush.Now there are two transaction in double buffer one for purge and other for updating deep clean. Some time snapshot can be present in the cache and sometime it doesn’t. When it has snapshot in cache, it causes snapshot chain corruption because both of the transaction will succeed.
In this change, we are invalidating the cache entry in
OMSnapshotPurgeRequestso that other APIs don't see the snapshot.What is the link to the Apache JIRA
HDDS-10524
How was this patch tested?
Wrote a test to mimic the same behavior as in described in the details. This test was run against existing code and with fix. It failed all the time in the existing code and passed in fixed code.
Also updated existing unit test to validate before and after snapshot purge transaction is committed.