HDDS-13452. Prevent snapshot defrag from happening before upgrade finalization by swamirishi · Pull Request #9240 · apache/ozone

swamirishi · 2025-11-03T04:11:41Z

What changes were proposed in this pull request?

Any snapshot defragmentation write operation can only happen after OM upgrade is finalized.
However we would still allow creation of the local yaml file which is inconsequential to the upgrade finalization since rollback would just mean that the yaml file would become orphan and can be deleted.
On OmSnapshotLocalDataManager initialization if the upgrade has not been finalized the snapshots which miss the yaml file are implicitly created by reading the rocksdb present in the version 0 path and getting the sst files from it for all the active snapshots. For the deleted snapshots the rocksdb could be in a partial purged state which could lead to issues while trying to open the snapshot rocksdb. Moreover the sstFileList fetch would not be useful since we would never defrag a snapshot when the previous snapshot is deleted or the snapshot itself has been deleted.
This is built on top of PR #9235 and #9150

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-13452

How was this patch tested?

Additional unit tests

Change-Id: Iba47aeb21663dfa407ab71339cef02c0d74b49f2

Change-Id: Ifd2feca1fddb144e4955db025f0b15a2ab1f3bfe

Change-Id: I34536ff06efb7d5a4942853f0fd83942ab398b5f

…otLocalDataManager Change-Id: I34536ff06efb7d5a4942853f0fd83942ab398b5f

Change-Id: I32bcaf2a1fb290f1790c02872a0230cd65586636

Change-Id: I105a2e8178c0444d52de41b99801f4ceb6d57ffd # Conflicts: # hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/util/Checksum.java # hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/util/ObjectSerializer.java # hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/util/YamlSerializer.java # hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmSnapshotLocalDataYaml.java # hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/OmSnapshotLocalDataManager.java # hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/TestOmSnapshotLocalDataYaml.java

Change-Id: I985170e38fb8beeb784048e85a08a4c79e1aec97

Change-Id: I33e6e6e825bf23c323ad7ed593d800a11720fa4f

Change-Id: If30b2c766db82adde72145c8ecd3e590ef54cc2d

Change-Id: Id3f2c49050bc3476b9e0f5f51dacb6d9acc4c2f7

Change-Id: I432960725b4c6c55aa906b5780cc3027e41e10db

Change-Id: I3c5514e5bbd251a2b5297d8f074cfde5c71fa543

Change-Id: Ib5a9e6c91bdccba17820263c47eaf2c8400e930d

Change-Id: Ica36e0615c7bc6aa9b6a7f6fafafd0f830d4bafb

Change-Id: I26b66f266bb7677e4b1078f5fcd9f2ce3a651a70 # Conflicts: # hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/OmSnapshotLocalDataManager.java

Change-Id: I1d93dbc048a42cc55ff1f8ffa420e52f967527b8

Change-Id: I34202928a7a367dd0a1e57219317ff34de352b78

Change-Id: Iad6f26cb71ec921c51ee2d138745df1a2663533f

Change-Id: Ic5f7e249cfb9cb3973cbcd4abd36b22a6ff8f5aa

…calDataProvider Change-Id: I3a004b4b435075a4348960aeed642e8da71e7e72

Change-Id: I06990bc9ab8fc7e1eb7bec255646a650bd8c35fe

Change-Id: I4c6c61c83aa9fadab8ecef854b99dcc0a89a2208

Change-Id: I0e476322372a302572f1fe79cbf2e874bfeac2ed

Change-Id: I31004e0c95dad64411c6fe848501a82f2f773cba

Change-Id: Id317c8b56e8b25c122b68eaf96599b9690d08f79

Change-Id: I3849387d064e093634e69cdaf870d27c1934cda5 # Conflicts: # hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/util/ObjectSerializer.java # hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/util/YamlSerializer.java # hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmSnapshotLocalData.java # hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmSnapshotLocalDataYaml.java # hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmSnapshotManager.java # hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/OmSnapshotLocalDataManager.java

Change-Id: Ie5e5f3dab4324103e8855dd15619d7755f0422e6

Change-Id: I55bd5c3ef7fc32910a9111328638de2edffcd541

Change-Id: I880997d3eebdf378f14c203c61c2d63b2d17552e

Change-Id: I13ba8e2fd012a3c964d657e83496c93a4f55a3be

…alization Change-Id: I26ac4e914906fd1e764877577afb9a51a489b07f

Change-Id: If064b64d1dfd624324090a946a33eab55535f28f

Change-Id: I3ed6575a96a12c349bb229fb3772c256a986f242

Change-Id: I83628385c1311ab9dc9de105ff29a0421fc8c4b3

Change-Id: I93602a49fc217d6167ed9629b6ff3ffcaccc8afa

Change-Id: Ic6fa9707da609617c187247efa03543e555ee181 # Conflicts: # hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmSnapshotManager.java

jojochuang · 2025-11-03T18:46:39Z

...ne-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/OmSnapshotLocalDataManager.java

    RDBStore store = (RDBStore) omMetadataManager.getStore();
    String checkpointPrefix = store.getDbLocation().getName();
    File snapshotDir = new File(store.getSnapshotsParentDir());
+    boolean upgradeNeeded = !layoutVersionManager.isAllowed(SNAPSHOT_DEFRAG);


software is updated, but not finalized yet.

...ne-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/OmSnapshotLocalDataManager.java

hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/SnapshotDefragService.java

Change-Id: Ied3c728c1f566bdd9327b6b5340892bed993cbf6

Change-Id: I1105c70ad060e8a344d042eadc3fa65b1469b716

Change-Id: Idc77f8a9a9a6cecb43b69d5532cf1bb2c679ce78

Change-Id: Ie6f1f79674be95db37a77969d3a06bce023e4bef

Change-Id: I1902394b0edfa8fa2f153787de98f21b20c55e59 # Conflicts: # hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmSnapshotManager.java # hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/OmSnapshotLocalDataManager.java # hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/snapshot/TestOmSnapshotLocalDataManager.java

swamirishi · 2025-11-08T19:22:30Z

@jojochuang can you review #9235 before this PR

Change-Id: I85d6072d0826ccc7f81a2579af307bfad4f049a7

Change-Id: Idabbe831cff0b00a51d32c2149765a09eecf7e23 # Conflicts: # hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmMetadataManagerImpl.java # hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmSnapshotManager.java

...e/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/service/TestKeyDeletingService.java

jojochuang

lgtm except for the System.out.println

jojochuang · 2025-11-13T00:08:08Z

hadoop-hdds/framework/src/test/java/org/apache/hadoop/hdds/utils/MapBackedTableIterator.java

  private Iterator<Table.KeyValue<String, V>> itr;
  private final String prefix;
-  private final TreeMap<String, V> values;
+  private final NavigableMap<String, V> values;


use the abstract parent class. Nice to have but not required in this PR.

Co-authored-by: Wei-Chiu Chuang <jojochuang@gmail.com>

adoroszlai · 2025-11-13T10:56:05Z

lgtm except for the System.out.println

If the patch needs to be updated before merge, then please:

don't approve, since it's confusing
don't trigger CI (by marking ready for review), since new run is needed anyway

adoroszlai · 2025-11-13T10:57:14Z

...op-hdds/framework/src/test/java/org/apache/hadoop/hdds/utils/db/StringInMemoryTestTable.java

+/**
+ *
+ */


Done I have added java doc here.

https://github.com/swamirishi/ozone/blob/HDDS-13452/hadoop-hdds/framework/src/test/java/org/apache/hadoop/hdds/utils/db/StringInMemoryTestTable.java

Change-Id: I66f8dde36164036193bf6c399f7c33c226ad6f2f

Change-Id: I4a49a827d2a3ee46682dcb9d2b673cc2c0f1c215

Change-Id: I1a2bbe2f55adab3fff553229cc58c0d08ac2bd3d

swamirishi · 2025-11-13T16:27:55Z

thank you for reviewing the patch @jojochuang and @adoroszlai for removing the debug print statement

swamirishi added 30 commits October 8, 2025 09:39

HDDS-13765. SnapshotLocalData yaml should also track snapshotId

6e241f6

Change-Id: Iba47aeb21663dfa407ab71339cef02c0d74b49f2

HDDS-13627. In memory Manager for Snapshot Local Data

a869500

Change-Id: Ifd2feca1fddb144e4955db025f0b15a2ab1f3bfe

HDDS-13627. In memory Manager for Snapshot Local Data

252d338

Change-Id: I34536ff06efb7d5a4942853f0fd83942ab398b5f

HDDS-13767. Refactor SnapshotLocalDataYaml related code into OmSnapsh…

4099bc6

…otLocalDataManager Change-Id: I34536ff06efb7d5a4942853f0fd83942ab398b5f

HDDS-13767. Fix pmd

e02670c

Change-Id: I32bcaf2a1fb290f1790c02872a0230cd65586636

HDDS-13627. Fix checkstyle

79580e9

Change-Id: I985170e38fb8beeb784048e85a08a4c79e1aec97

Merge remote-tracking branch 'apache/master' into HEAD

2a331ef

Change-Id: I33e6e6e825bf23c323ad7ed593d800a11720fa4f

Merge remote-tracking branch 'apache/master' into HEAD

c4f69e2

Change-Id: If30b2c766db82adde72145c8ecd3e590ef54cc2d

HDDS-13627. Add tests

afbc592

Change-Id: Id3f2c49050bc3476b9e0f5f51dacb6d9acc4c2f7

HDDS-13783. Implement locks for OmSnapshotLocalDataManager

70ac2c7

Change-Id: I432960725b4c6c55aa906b5780cc3027e41e10db

HDDS-13783. Implement locks for OmSnapshotLocalDataManager

b554cc7

Change-Id: I3c5514e5bbd251a2b5297d8f074cfde5c71fa543

HDDS-13783. Refactor inline variable

49eccfa

Change-Id: Ib5a9e6c91bdccba17820263c47eaf2c8400e930d

HDDS-13627. Refactor map data structure

51eda04

Change-Id: Ica36e0615c7bc6aa9b6a7f6fafafd0f830d4bafb

Merge remote-tracking branch 'origin/HDDS-13627' into HEAD

25f766c

Change-Id: I26b66f266bb7677e4b1078f5fcd9f2ce3a651a70 # Conflicts: # hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/OmSnapshotLocalDataManager.java

HDDS-13783. Add more condition to upsert

96689fa

Change-Id: I1d93dbc048a42cc55ff1f8ffa420e52f967527b8

HDDS-13783. Add java doc comment

0674299

Change-Id: I34202928a7a367dd0a1e57219317ff34de352b78

HDDS-13783. Add java doc comment

5d9fc49

Change-Id: Iad6f26cb71ec921c51ee2d138745df1a2663533f

HDDS-13783. Implement full lock

2d88176

Change-Id: Ic5f7e249cfb9cb3973cbcd4abd36b22a6ff8f5aa

HDDS-13783. Refactor and move modify method into WritableOmSnapshotLo…

a3c4c69

…calDataProvider Change-Id: I3a004b4b435075a4348960aeed642e8da71e7e72

HDDS-13783. Make full lock non static

686d0c7

Change-Id: I06990bc9ab8fc7e1eb7bec255646a650bd8c35fe

HDDS-13783. Fix remove

491a54b

Change-Id: I4c6c61c83aa9fadab8ecef854b99dcc0a89a2208

HDDS-13627. Fix findbugs

5e69ee9

Change-Id: I0e476322372a302572f1fe79cbf2e874bfeac2ed

HDDS-13785. Remove orphan versions from SnapshotLocalData Yaml file

d36622a

Change-Id: I31004e0c95dad64411c6fe848501a82f2f773cba

HDDS-13785. Fix findbugs

ee213d1

Change-Id: Id317c8b56e8b25c122b68eaf96599b9690d08f79

HDDS-13627. Fix tests

a95604e

Change-Id: Ie5e5f3dab4324103e8855dd15619d7755f0422e6

HDDS-13627. remove checksum interface

5a90fcf

Change-Id: I55bd5c3ef7fc32910a9111328638de2edffcd541

HDDS-13627. Fix test failures

20d7d6a

Change-Id: I880997d3eebdf378f14c203c61c2d63b2d17552e

HDDS-13785. Set defrag flag on previous snapshotId update

ae655cb

Change-Id: I13ba8e2fd012a3c964d657e83496c93a4f55a3be

HDDS-13452. Prevent snapshot defrag from happening before upgrade fin…

bb5139f

…alization Change-Id: I26ac4e914906fd1e764877577afb9a51a489b07f

swamirishi requested review from jojochuang and smengcl November 3, 2025 04:11

swamirishi added the snapshot https://issues.apache.org/jira/browse/HDDS-6517 label Nov 3, 2025

swamirishi added 5 commits November 3, 2025 00:27

Merge remote-tracking branch 'apache/master' into HEAD

c46e1ae

Change-Id: If064b64d1dfd624324090a946a33eab55535f28f

HDDS-13849. Fix test

298ae36

Change-Id: I3ed6575a96a12c349bb229fb3772c256a986f242

Merge remote-tracking branch 'origin/HDDS-13849' into HEAD

8e43b90

Change-Id: I83628385c1311ab9dc9de105ff29a0421fc8c4b3

Merge remote-tracking branch 'apache/master' into HEAD

55c68bd

Change-Id: I93602a49fc217d6167ed9629b6ff3ffcaccc8afa

Merge remote-tracking branch 'apache/master' into HEAD

d1d04cf

Change-Id: Ic6fa9707da609617c187247efa03543e555ee181 # Conflicts: # hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmSnapshotManager.java

jojochuang reviewed Nov 3, 2025

View reviewed changes

swamirishi added 5 commits November 5, 2025 13:50

HDDS-13785. Address review comments

018571c

Change-Id: Ied3c728c1f566bdd9327b6b5340892bed993cbf6

HDDS-13785. Address review comments

6cd54dd

Change-Id: I1105c70ad060e8a344d042eadc3fa65b1469b716

HDDS-13785. Remove unnecessary read lock

261a669

Change-Id: Idc77f8a9a9a6cecb43b69d5532cf1bb2c679ce78

Merge remote-tracking branch 'origin/HDDS-13785' into HEAD

8955d80

Change-Id: Ie6f1f79674be95db37a77969d3a06bce023e4bef

swamirishi added 2 commits November 10, 2025 15:38

Merge remote-tracking branch 'apache/master' into HEAD

5823347

Change-Id: I85d6072d0826ccc7f81a2579af307bfad4f049a7

swamirishi force-pushed the HDDS-13452 branch from 5e61924 to 260942e Compare November 11, 2025 02:28

jojochuang reviewed Nov 12, 2025

View reviewed changes

...e/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/service/TestKeyDeletingService.java Outdated Show resolved Hide resolved

jojochuang approved these changes Nov 13, 2025

View reviewed changes

jojochuang marked this pull request as ready for review November 13, 2025 07:13

Remove leftover debug comment

c88c9bf

Co-authored-by: Wei-Chiu Chuang <jojochuang@gmail.com>

adoroszlai reviewed Nov 13, 2025

View reviewed changes

swamirishi added 3 commits November 13, 2025 06:05

Merge remote-tracking branch 'apache/master' into HEAD

39605f3

Change-Id: I66f8dde36164036193bf6c399f7c33c226ad6f2f

HDDS-13452. Address review comments

ae35dc9

Change-Id: I4a49a827d2a3ee46682dcb9d2b673cc2c0f1c215

Merge remote-tracking branch 'origin/HDDS-13452' into HEAD

a7daebd

Change-Id: I1a2bbe2f55adab3fff553229cc58c0d08ac2bd3d

swamirishi merged commit 47473dc into apache:master Nov 13, 2025
43 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HDDS-13452. Prevent snapshot defrag from happening before upgrade finalization#9240

HDDS-13452. Prevent snapshot defrag from happening before upgrade finalization#9240
swamirishi merged 170 commits intoapache:masterfrom
swamirishi:HDDS-13452

swamirishi commented Nov 3, 2025 •

edited

Loading

Uh oh!

jojochuang Nov 3, 2025

Uh oh!

Uh oh!

Uh oh!

swamirishi commented Nov 8, 2025

Uh oh!

Uh oh!

jojochuang left a comment

Uh oh!

jojochuang Nov 13, 2025

Uh oh!

adoroszlai commented Nov 13, 2025

Uh oh!

adoroszlai Nov 13, 2025

Uh oh!

swamirishi Nov 13, 2025

Uh oh!

swamirishi Nov 13, 2025

Uh oh!

Uh oh!

swamirishi commented Nov 13, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

swamirishi commented Nov 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

What is the link to the Apache JIRA

How was this patch tested?

Uh oh!

jojochuang Nov 3, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

swamirishi commented Nov 8, 2025

Uh oh!

Uh oh!

jojochuang left a comment

Choose a reason for hiding this comment

Uh oh!

jojochuang Nov 13, 2025

Choose a reason for hiding this comment

Uh oh!

adoroszlai commented Nov 13, 2025

Uh oh!

adoroszlai Nov 13, 2025

Choose a reason for hiding this comment

Uh oh!

swamirishi Nov 13, 2025

Choose a reason for hiding this comment

Uh oh!

swamirishi Nov 13, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

swamirishi commented Nov 13, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

swamirishi commented Nov 3, 2025 •

edited

Loading