HDDS-13830. Snapshot Rocks DB directory path computation based on local version of snapshot #9208

swamirishi · 2025-10-28T10:29:09Z

What changes were proposed in this pull request?

Snapshot Rocks DB directory path needs to be computed based on local version of snapshot. Currently snapshot rocksdb path is /db.snapshot/om.db-<snapshotId> however whenever a snapshot gets defragged the snapshot would be written to a new path and the version in the yaml file would be updated to atomically point the snapshot to the newly created rocksdb instance atomically.

Thus snapshot rocksdb path would become /db.snapshot/om.db-<snapshotId>-<version>

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-13830

How was this patch tested?

Adding additional unit test for the getSnapshotPath function

Change-Id: Iba47aeb21663dfa407ab71339cef02c0d74b49f2

Change-Id: Ifd2feca1fddb144e4955db025f0b15a2ab1f3bfe

Change-Id: I34536ff06efb7d5a4942853f0fd83942ab398b5f

…otLocalDataManager Change-Id: I34536ff06efb7d5a4942853f0fd83942ab398b5f

Change-Id: I32bcaf2a1fb290f1790c02872a0230cd65586636

Change-Id: I105a2e8178c0444d52de41b99801f4ceb6d57ffd # Conflicts: # hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/util/Checksum.java # hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/util/ObjectSerializer.java # hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/util/YamlSerializer.java # hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmSnapshotLocalDataYaml.java # hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/OmSnapshotLocalDataManager.java # hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/TestOmSnapshotLocalDataYaml.java

Change-Id: I985170e38fb8beeb784048e85a08a4c79e1aec97

Change-Id: I33e6e6e825bf23c323ad7ed593d800a11720fa4f

Change-Id: If30b2c766db82adde72145c8ecd3e590ef54cc2d

Change-Id: Id3f2c49050bc3476b9e0f5f51dacb6d9acc4c2f7

Change-Id: I432960725b4c6c55aa906b5780cc3027e41e10db

Change-Id: I3c5514e5bbd251a2b5297d8f074cfde5c71fa543

Change-Id: Ib5a9e6c91bdccba17820263c47eaf2c8400e930d

Change-Id: Ica36e0615c7bc6aa9b6a7f6fafafd0f830d4bafb

Change-Id: I26b66f266bb7677e4b1078f5fcd9f2ce3a651a70 # Conflicts: # hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/OmSnapshotLocalDataManager.java

Change-Id: I1d93dbc048a42cc55ff1f8ffa420e52f967527b8

Change-Id: I34202928a7a367dd0a1e57219317ff34de352b78

Change-Id: Iad6f26cb71ec921c51ee2d138745df1a2663533f

Change-Id: Ic5f7e249cfb9cb3973cbcd4abd36b22a6ff8f5aa

…calDataProvider Change-Id: I3a004b4b435075a4348960aeed642e8da71e7e72

Change-Id: I06990bc9ab8fc7e1eb7bec255646a650bd8c35fe

Change-Id: I4c6c61c83aa9fadab8ecef854b99dcc0a89a2208

Change-Id: I0e476322372a302572f1fe79cbf2e874bfeac2ed

Change-Id: I3849387d064e093634e69cdaf870d27c1934cda5 # Conflicts: # hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/util/ObjectSerializer.java # hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/util/YamlSerializer.java # hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmSnapshotLocalData.java # hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmSnapshotLocalDataYaml.java # hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmSnapshotManager.java # hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/OmSnapshotLocalDataManager.java

Change-Id: Ie5e5f3dab4324103e8855dd15619d7755f0422e6

Change-Id: I55bd5c3ef7fc32910a9111328638de2edffcd541

Change-Id: I880997d3eebdf378f14c203c61c2d63b2d17552e

Change-Id: I20812cdf61886e55b4ead6003566e1d72db77f4a

Change-Id: I02de81771c9102f1212bf1962e65095910ab8207

Change-Id: I8360183ef8ac68a95a05a6a2b00bb7ede5d57d12

Change-Id: I1bb9832e7e8c40deeccb9d0868eaf5772f39b7f9

Change-Id: I59bd793796f77e88f44e7333fbb01ffabd477f67

Change-Id: Iec8a011ac2b2dc369be5ec1d659be3a0f580890a # Conflicts: # hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/SnapshotDefragService.java # hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/OmSnapshotLocalDataManager.java # hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/snapshot/TestOmSnapshotLocalDataManager.java

Change-Id: Idb8e173737fd496d2d36e2b9409e5b7e1f45e995

smengcl · 2025-11-01T00:02:44Z

The PR description looks incomplete to me. Can you check?

smengcl · 2025-11-01T00:04:18Z

In the SDS PR I actually was dealing with a similar problem. Tho I simply appended local defrag version to the DB checkpoitn dir name. A slight change in OmMetadataManagerImpl: https://github.com/apache/ozone/pull/9227/files

swamirishi · 2025-11-01T03:55:48Z

In the SDS PR I actually was dealing with a similar problem. Tho I simply appended local defrag version to the DB checkpoitn dir name. A slight change in OmMetadataManagerImpl: https://github.com/apache/ozone/pull/9227/files

I wanted this to be a more structural fix thus made the change in the path computation path itself so that it automatically reflects everywhere

Change-Id: I39e1d4a1f504241a2ebbfb5b36c84e887512c8ed

swamirishi · 2025-11-01T03:58:56Z

The PR description looks incomplete to me. Can you check?

Fixed the issue < character had to be escaped

Change-Id: I88f541c712d847515f6ef5b1d094cc5168936966

Change-Id: I71e2210c6398aa9b64f93137bb5655e1c41026e3

swamirishi · 2025-11-01T05:06:17Z

@smengcl are you good with the changes can this be merged?

Change-Id: I5541c7f931bb0b60af238102b5afc8086be53a6c

Change-Id: I7077460d4b9d87d460244e3df51d51f078a83970

Change-Id: If9cd8e82083f90997d7ce052408b33648d9b2c2a

jojochuang

Looks good. How does it handle backward compatibility? (i.e. snapshots created before the snapshot defrag is implemented, and the checkpoint dir does not include version id)

swamirishi · 2025-11-02T12:26:20Z

Looks good. How does it handle backward compatibility? (i.e. snapshots created before the snapshot defrag is implemented, and the checkpoint dir does not include version id)

We would be handling upgrade flow by adding a new layout version. For the older snapshots we can always implicitly create the yaml files for the entries which don't have the yaml file if and only if the upgrade is not finalized. When the upgrade is finalized that is when we should have the yaml file already present and we should throw an exception if it is not there. For the deleted snapshot we need not track the sst file since it could be the case that the snapshot could have been purged.

Change-Id: I011b0b80bc646c94b422ab808935de4fbee2bf30 # Conflicts: # hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestOMDbCheckpointServletInodeBasedXfer.java # hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OMDBCheckpointServletInodeBasedXfer.java

Change-Id: Ia02449ab5237ddb6021fa3a62ed66290c1dff83d

swamirishi · 2025-11-02T19:18:08Z

@jojochuang can we merge this issue we can handle the upgrade scenario under a different jira

jojochuang

LGTM

swamirishi added 30 commits October 8, 2025 09:39

HDDS-13765. SnapshotLocalData yaml should also track snapshotId

6e241f6

Change-Id: Iba47aeb21663dfa407ab71339cef02c0d74b49f2

HDDS-13627. In memory Manager for Snapshot Local Data

a869500

Change-Id: Ifd2feca1fddb144e4955db025f0b15a2ab1f3bfe

HDDS-13627. In memory Manager for Snapshot Local Data

252d338

Change-Id: I34536ff06efb7d5a4942853f0fd83942ab398b5f

HDDS-13767. Refactor SnapshotLocalDataYaml related code into OmSnapsh…

4099bc6

…otLocalDataManager Change-Id: I34536ff06efb7d5a4942853f0fd83942ab398b5f

HDDS-13767. Fix pmd

e02670c

Change-Id: I32bcaf2a1fb290f1790c02872a0230cd65586636

HDDS-13627. Fix checkstyle

79580e9

Change-Id: I985170e38fb8beeb784048e85a08a4c79e1aec97

Merge remote-tracking branch 'apache/master' into HEAD

2a331ef

Change-Id: I33e6e6e825bf23c323ad7ed593d800a11720fa4f

Merge remote-tracking branch 'apache/master' into HEAD

c4f69e2

Change-Id: If30b2c766db82adde72145c8ecd3e590ef54cc2d

HDDS-13627. Add tests

afbc592

Change-Id: Id3f2c49050bc3476b9e0f5f51dacb6d9acc4c2f7

HDDS-13783. Implement locks for OmSnapshotLocalDataManager

70ac2c7

Change-Id: I432960725b4c6c55aa906b5780cc3027e41e10db

HDDS-13783. Implement locks for OmSnapshotLocalDataManager

b554cc7

Change-Id: I3c5514e5bbd251a2b5297d8f074cfde5c71fa543

HDDS-13783. Refactor inline variable

49eccfa

Change-Id: Ib5a9e6c91bdccba17820263c47eaf2c8400e930d

HDDS-13627. Refactor map data structure

51eda04

Change-Id: Ica36e0615c7bc6aa9b6a7f6fafafd0f830d4bafb

Merge remote-tracking branch 'origin/HDDS-13627' into HEAD

25f766c

Change-Id: I26b66f266bb7677e4b1078f5fcd9f2ce3a651a70 # Conflicts: # hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/OmSnapshotLocalDataManager.java

HDDS-13783. Add more condition to upsert

96689fa

Change-Id: I1d93dbc048a42cc55ff1f8ffa420e52f967527b8

HDDS-13783. Add java doc comment

0674299

Change-Id: I34202928a7a367dd0a1e57219317ff34de352b78

HDDS-13783. Add java doc comment

5d9fc49

Change-Id: Iad6f26cb71ec921c51ee2d138745df1a2663533f

HDDS-13783. Implement full lock

2d88176

Change-Id: Ic5f7e249cfb9cb3973cbcd4abd36b22a6ff8f5aa

HDDS-13783. Refactor and move modify method into WritableOmSnapshotLo…

a3c4c69

…calDataProvider Change-Id: I3a004b4b435075a4348960aeed642e8da71e7e72

HDDS-13783. Make full lock non static

686d0c7

Change-Id: I06990bc9ab8fc7e1eb7bec255646a650bd8c35fe

HDDS-13783. Fix remove

491a54b

Change-Id: I4c6c61c83aa9fadab8ecef854b99dcc0a89a2208

HDDS-13627. Fix findbugs

5e69ee9

Change-Id: I0e476322372a302572f1fe79cbf2e874bfeac2ed

HDDS-13627. Fix tests

a95604e

Change-Id: Ie5e5f3dab4324103e8855dd15619d7755f0422e6

HDDS-13627. remove checksum interface

5a90fcf

Change-Id: I55bd5c3ef7fc32910a9111328638de2edffcd541

HDDS-13627. Fix test failures

20d7d6a

Change-Id: I880997d3eebdf378f14c203c61c2d63b2d17552e

Merge remote-tracking branch 'origin/HDDS-13627' into HEAD

25fa6ae

Change-Id: I20812cdf61886e55b4ead6003566e1d72db77f4a

HDDS-13783. Fix findbugs

d419283

Change-Id: I02de81771c9102f1212bf1962e65095910ab8207

HDDS-13783. Fix pmd

8a44308

Change-Id: I8360183ef8ac68a95a05a6a2b00bb7ede5d57d12

swamirishi added 3 commits October 29, 2025 01:07

HDDS-13783. Add comments for localDataGraph

4596386

Change-Id: I1bb9832e7e8c40deeccb9d0868eaf5772f39b7f9

Merge remote-tracking branch 'origin/HDDS-13783' into HEAD

25ee4e2

Change-Id: I59bd793796f77e88f44e7333fbb01ffabd477f67

swamirishi requested review from jojochuang, sadanand48 and smengcl October 29, 2025 12:07

Merge remote-tracking branch 'apache/master' into HEAD

e58ff09

Change-Id: Idb8e173737fd496d2d36e2b9409e5b7e1f45e995

Merge remote-tracking branch 'apache/master' into HEAD

408e213

Change-Id: I39e1d4a1f504241a2ebbfb5b36c84e887512c8ed

swamirishi marked this pull request as ready for review November 1, 2025 03:59

swamirishi added 2 commits November 1, 2025 01:04

HDDS-13830. Fix test

c829a8b

Change-Id: I88f541c712d847515f6ef5b1d094cc5168936966

Merge remote-tracking branch 'apache/master' into HEAD

8e91e47

Change-Id: I71e2210c6398aa9b64f93137bb5655e1c41026e3

HDDS-13830. Fix test

bfd341c

Change-Id: I5541c7f931bb0b60af238102b5afc8086be53a6c

swamirishi marked this pull request as draft November 1, 2025 11:33

swamirishi added 2 commits November 1, 2025 07:35

HDDS-13830. Fix test

4ccd3fc

Change-Id: I7077460d4b9d87d460244e3df51d51f078a83970

HDDS-13830. Fix pmd

41b7cfb

Change-Id: If9cd8e82083f90997d7ce052408b33648d9b2c2a

jojochuang reviewed Nov 2, 2025

View reviewed changes

jojochuang marked this pull request as ready for review November 2, 2025 09:16

swamirishi added 2 commits November 2, 2025 13:56

HDDS-13830. Fix mrge issue

d0422ae

Change-Id: Ia02449ab5237ddb6021fa3a62ed66290c1dff83d

swamirishi requested a review from jojochuang November 2, 2025 21:48

jojochuang approved these changes Nov 3, 2025

View reviewed changes

jojochuang merged commit 5c35ebb into apache:master Nov 3, 2025
43 checks passed

smengcl mentioned this pull request Nov 3, 2025

HDDS-13009. Background snapshot defrag service #9227

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HDDS-13830. Snapshot Rocks DB directory path computation based on local version of snapshot #9208

HDDS-13830. Snapshot Rocks DB directory path computation based on local version of snapshot #9208

Uh oh!

swamirishi commented Oct 28, 2025 •

edited

Loading

Uh oh!

smengcl commented Nov 1, 2025

Uh oh!

smengcl commented Nov 1, 2025

Uh oh!

swamirishi commented Nov 1, 2025

Uh oh!

swamirishi commented Nov 1, 2025

Uh oh!

swamirishi commented Nov 1, 2025

Uh oh!

jojochuang left a comment

Uh oh!

swamirishi commented Nov 2, 2025 •

edited

Loading

Uh oh!

swamirishi commented Nov 2, 2025

Uh oh!

jojochuang left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

HDDS-13830. Snapshot Rocks DB directory path computation based on local version of snapshot #9208

HDDS-13830. Snapshot Rocks DB directory path computation based on local version of snapshot #9208

Uh oh!

Conversation

swamirishi commented Oct 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

What is the link to the Apache JIRA

How was this patch tested?

Uh oh!

smengcl commented Nov 1, 2025

Uh oh!

smengcl commented Nov 1, 2025

Uh oh!

swamirishi commented Nov 1, 2025

Uh oh!

swamirishi commented Nov 1, 2025

Uh oh!

swamirishi commented Nov 1, 2025

Uh oh!

jojochuang left a comment

Choose a reason for hiding this comment

Uh oh!

swamirishi commented Nov 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

swamirishi commented Nov 2, 2025

Uh oh!

jojochuang left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

swamirishi commented Oct 28, 2025 •

edited

Loading

swamirishi commented Nov 2, 2025 •

edited

Loading