HDDS-13859. OmSnapshotLocalDataManager should handle needsDefrag flag in the yaml file#9224
Merged
swamirishi merged 94 commits intoapache:masterfrom Oct 31, 2025
Merged
HDDS-13859. OmSnapshotLocalDataManager should handle needsDefrag flag in the yaml file#9224swamirishi merged 94 commits intoapache:masterfrom
swamirishi merged 94 commits intoapache:masterfrom
Conversation
Change-Id: Iba47aeb21663dfa407ab71339cef02c0d74b49f2
Change-Id: Ifd2feca1fddb144e4955db025f0b15a2ab1f3bfe
Change-Id: I34536ff06efb7d5a4942853f0fd83942ab398b5f
…otLocalDataManager Change-Id: I34536ff06efb7d5a4942853f0fd83942ab398b5f
Change-Id: I32bcaf2a1fb290f1790c02872a0230cd65586636
Change-Id: I105a2e8178c0444d52de41b99801f4ceb6d57ffd # Conflicts: # hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/util/Checksum.java # hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/util/ObjectSerializer.java # hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/util/YamlSerializer.java # hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmSnapshotLocalDataYaml.java # hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/OmSnapshotLocalDataManager.java # hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/TestOmSnapshotLocalDataYaml.java
Change-Id: I985170e38fb8beeb784048e85a08a4c79e1aec97
Change-Id: I33e6e6e825bf23c323ad7ed593d800a11720fa4f
Change-Id: If30b2c766db82adde72145c8ecd3e590ef54cc2d
Change-Id: Id3f2c49050bc3476b9e0f5f51dacb6d9acc4c2f7
Change-Id: I432960725b4c6c55aa906b5780cc3027e41e10db
Change-Id: I3c5514e5bbd251a2b5297d8f074cfde5c71fa543
Change-Id: Ib5a9e6c91bdccba17820263c47eaf2c8400e930d
Change-Id: Ica36e0615c7bc6aa9b6a7f6fafafd0f830d4bafb
Change-Id: I26b66f266bb7677e4b1078f5fcd9f2ce3a651a70 # Conflicts: # hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/OmSnapshotLocalDataManager.java
Change-Id: I1d93dbc048a42cc55ff1f8ffa420e52f967527b8
Change-Id: I34202928a7a367dd0a1e57219317ff34de352b78
Change-Id: Iad6f26cb71ec921c51ee2d138745df1a2663533f
Change-Id: Ic5f7e249cfb9cb3973cbcd4abd36b22a6ff8f5aa
…calDataProvider Change-Id: I3a004b4b435075a4348960aeed642e8da71e7e72
Change-Id: I06990bc9ab8fc7e1eb7bec255646a650bd8c35fe
Change-Id: I4c6c61c83aa9fadab8ecef854b99dcc0a89a2208
Change-Id: I0e476322372a302572f1fe79cbf2e874bfeac2ed
Change-Id: I3849387d064e093634e69cdaf870d27c1934cda5 # Conflicts: # hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/util/ObjectSerializer.java # hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/util/YamlSerializer.java # hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmSnapshotLocalData.java # hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmSnapshotLocalDataYaml.java # hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmSnapshotManager.java # hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/OmSnapshotLocalDataManager.java
Change-Id: Ie5e5f3dab4324103e8855dd15619d7755f0422e6
Change-Id: I55bd5c3ef7fc32910a9111328638de2edffcd541
Change-Id: I880997d3eebdf378f14c203c61c2d63b2d17552e
Change-Id: I20812cdf61886e55b4ead6003566e1d72db77f4a
Change-Id: I02de81771c9102f1212bf1962e65095910ab8207
Change-Id: I8360183ef8ac68a95a05a6a2b00bb7ede5d57d12
Change-Id: I37d2f5c07f405f3069a4cb99881d5d1e67110e79
Change-Id: I119e8cede140c755d3cd09c0a56234ff4906be98
Change-Id: I6bbcdac1acb389ee3c08efb37676cccdb3d783c6
… the value on SnapshotPurgeRequest Change-Id: I0e9de1954fc65a907b9f4a98d6f84b2772d15551
Change-Id: I36c48585d17042491f7dc61a84845c999802d1fa
Change-Id: I2f7cb5ec772d2d2de99e8d3cba8bdc34e3f36efd
Change-Id: Ida628ebab658b2a31fe5d654f19035e96f3163ac
Change-Id: I960fd1d25e0f323d56f5e88ffb03261d777d9021
Change-Id: I1bb9832e7e8c40deeccb9d0868eaf5772f39b7f9
Change-Id: Ia1003e1bb5b4b891c36f5471a3f6d160d595fd27
Change-Id: I694d2512e2c1f2f8f81ab088f7cd0295efef6ea1 # Conflicts: # hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/OmSnapshotLocalDataManager.java # hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/TestOmSnapshotLocalDataYaml.java # hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/snapshot/TestOmSnapshotLocalDataManager.java
Change-Id: Ic484c5477fa866299af43a37031412f269f90c2a
Change-Id: I90e886ac5ca954cdf4fa7110b175168b2facac4b
… in the yaml file Change-Id: Id4e0082b42ded19a5b05418555478c5a71ae2d1a
Change-Id: Ib1ab4c81e920f1465c3e9046092cdbf0727bcf7d
Change-Id: Iaea152b0b69dcdd52a6f3806c713ee2b48d3875c
Change-Id: I79190cbc72c56677600386477d25719bd2fda3b2
Change-Id: Ie531cc345a2fe4d9945837fbe05dd9171cfcaec7 # Conflicts: # hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/TestOmSnapshotManager.java
Change-Id: I6527b98cd16850079b2f3855f4bfb5a5b59fa3ee
jojochuang
approved these changes
Oct 31, 2025
Contributor
jojochuang
left a comment
jojochuang
left a comment
There was a problem hiding this comment.
LGTM. But I think the defrag conditions is not mentioned in the design doc. We should revise the design doc after it's all done.
| } | ||
| } | ||
|
|
||
| public void addSnapshotVersion(RDBStore snapshotStore) throws IOException { |
| List<UUID> snapshotIds = createSnapshotLocalData(localDataManager, 2); | ||
| for (UUID snapshotId : snapshotIds) { | ||
| try (ReadableOmSnapshotLocalDataProvider snap = localDataManager.getOmSnapshotLocalData(snapshotId)) { | ||
| assertTrue(snap.needsDefrag()); |
Contributor
There was a problem hiding this comment.
newly created snapshots needs to defrag
| } | ||
| addVersionsToLocalData(localDataManager, snapshotIds.get(1), ImmutableMap.of(1, 3, 2, previousVersion)); | ||
| try (ReadableOmSnapshotLocalDataProvider snap = localDataManager.getOmSnapshotLocalData(snapshotIds.get(1))) { | ||
| assertEquals(previousVersion < snap.getPreviousSnapshotLocalData().getVersion(), snap.needsDefrag()); |
Contributor
There was a problem hiding this comment.
when the previousSnapshotId in the yaml file is updated
Contributor
Author
|
Thank you for reviewing the patch @jojochuang |
Contributor
|
Thanks @swamirishi for the quick fix! |
Contributor
|
@swamirishi please set fix version when resolving Jira issue after PR merge |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
OmSnapshotLocalDataManager should update needsDefrag flag in the yaml file based on the operations performed on the SnapshotLocalDataYaml.
Needs Defrag flag needs to be set when the snapshotLocalDataYaml was not present in the localDataManager before i.e. this is a newly created snapshot. The flag also needs to be set when the previousSnapshotId in the yaml file is updated which means there was a chain update inturn meaning the snapshot needs to be defragged.
The flag should be reset when a new version is added and a new version would be only added when a snapshot is defragged thus the flag should be reset.
To check whether a snapshot should be defragged either the flag should be set or the latest version of the previous snapshot is greater than the previous version the current snapshot's version is pointing to in the graph.
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-13859
This is built on PR #9193
How was this patch tested?
Added Unit Tests