HDDS-13180. Add replicatedSizeOfFiles to NSSummary #8568

tanvipenumudy · 2025-06-05T17:00:58Z

What changes were proposed in this pull request?

This change adds a persisted replicatedSizeOfFiles field to NSSummary to track the total expected replicated size of a directory's sub-files.
The value is dynamically updated upon PUT and DELETE key events.
The endpoint: /keys/deletePending/dirs is also updated to include the replicated size information.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-13180

How was this patch tested?

Added Unit and Integration Tests
Green CI: https://github.com/apache/ozone/actions/runs/15475444094/

Manual Testing:

bash-5.1$ ozone sh volume create vol1
bash-5.1$ ozone sh bucket create vol1/buck1 --layout=FILE_SYSTEM_OPTIMIZED
bash-5.1$ ozone fs -mkdir -p ofs://om/vol1/buck1/dir1/dir2/dir3

# Create 1 MB file:
bash-5.1$ dd if=/dev/zero of=test.txt bs=1M count=1
bash-5.1$ ozone fs -put test.txt ofs://om/vol1/buck1/dir1/file1
bash-5.1$ ozone fs -put test.txt ofs://om/vol1/buck1/dir1/dir2/file2
bash-5.1$ ozone fs -put test.txt ofs://om/vol1/buck1/dir1/dir2/dir3/file3

Directory structure:

dir1/
    ├── file1 (1 MB)
    └── dir2/
        ├── file2 (1 MB)
        └── dir3/
            └── file3 (1 MB)

# Recursively delete top-level directory:
bash-5.1$ ozone fs -rm -r ofs://om/vol1/buck1/dir1

GET /api/v1/keys/deletePending/dirs/:

{
  "lastKey": "/-9223372036854775552/-9223372036854775040/-9223372036854767870/250608184600/-9223372036854751487",
  "replicatedDataSize": 9437184,
  "unreplicatedDataSize": 3145728,
  "deletedDirInfo": [
    {
      "key": "250608184600",
      "path": "vol1/buck1/.Trash/hadoop/250608184600",
      "inStateSince": 1749408310990,
      "size": 3145728,
      "replicatedSize": 9437184,
      "replicationInfo": {
        "replicationFactor": "ONE",
        "requiredNodes": 1,
        "minimumNodes": 1,
        "replicationType": "RATIS"
      },
      "creationTime": 1749408437900,
      "modificationTime": 1749408437900,
      "isKey": false
    }
  ],
  "status": "OK"
}

Unreplicated size: 3145728 (3 MB)
Replicated size (RATIS THREE): 9437184 (9 MB)

devabhishekpal

Thanks @tanvipenumudy for this patch.
I think this might need an upgrade handler in case the user is upgrading Recon.
Currently we do not have a handler in the traditional way like it is present for other DerbyDB tables.

In this case you would have to trigger a fresh build of NSSummary after upgrade to ensure that the new changes are reflected.

@ArafatKhan2198 could you help out with how this can be done?

hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/OMDBInsightEndpoint.java

devabhishekpal

Hi @tanvipenumudy, overall the patch looks good to me.
Added a few minor comments.

The only thing missing right now is the upgrade handler for this patch.

@ArafatKhan2198 @devmadhuu could you help with taking a look at this patch as well?
Thanks

tanvipenumudy · 2025-07-09T06:12:20Z

TestReconInsightsForDeletedDirectoriesEC passes locally, performing a rerun!

devabhishekpal · 2025-07-09T07:13:36Z

Thanks for the changes @tanvipenumudy. Could you add tests for the upgrade handler as well?

sumitagrawl

LGTM

hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/OMDBInsightEndpoint.java

ChenSammi · 2025-07-14T08:03:04Z

.../src/main/java/org/apache/hadoop/ozone/recon/upgrade/ReplicatedSizeOfFilesUpgradeAction.java

+      ReconNamespaceSummaryManager nsSummaryManager = injector.getInstance(ReconNamespaceSummaryManager.class);
+      ReconOMMetadataManager omMetadataManager = injector.getInstance(ReconOMMetadataManager.class);
+      LOG.info("Starting full rebuild of NSSummary for REPLICATED_SIZE_OF_FILES upgrade...");
+      nsSummaryManager.rebuildNSSummaryTree(omMetadataManager);


@tanvipenumudy , what's the estimation time of this rebuild process for a Recon DB, size similar to some of user bigger users?

If the upgrade will take hours, is it accepted? @devmadhuu

I believe something similar was done (tree rebuild process was triggered) when the parentId field was newly introduced (PR: #6492). Not sure if we have any existing numbers on this cc: @ArafatKhan2198.

As per an offline discussion with @ArafatKhan2198, please find the stats below based on a previous benchmark:

Total Data: 1.347 TB

Directories: 5.7 million

Keys: 8.7 million

Tree Depth: 5-10 levels

The total time it took to (re)build the NSSummary tree was ~5 mins.

Thanks @tanvipenumudy for the info. The Recon DB is 1.347TB for 5.7m dirs and 8.7m keys? The DB size is just beyond the expectation.
Currently "nsSummaryManager.rebuildNSSummaryTree(omMetadataManager)" is a blocking call. It returns when the rebuild finished.

How about if we trigger rebuild as an async operation because we have plans to show the status update of recon tasks on Overview page if they are in progress. This may ease upgrade process efficient and quicker...

Thank you @devmadhuu, as discussed this change can be part of HDDS-13443!

The Recon DB is 1.347TB for 5.7m dirs and 8.7m keys? The DB size is just beyond the expectation.

@ChenSammi, I believe this is the total cluster capacity (please correct me if I'm wrong @ArafatKhan2198).

"nsSummaryManager.rebuildNSSummaryTree(omMetadataManager)" blocks hours during Recon update, will there be any impact to Recon's other part functionality?

Thank you @devmadhuu and @ArafatKhan2198 for clarifying the second half of the question!

hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/fs/ozone/TestLeaseRecovery.java

devmadhuu

@tanvipenumudy , can you check a comment pls ?

devmadhuu · 2025-07-17T07:42:52Z

.../src/main/java/org/apache/hadoop/ozone/recon/upgrade/ReplicatedSizeOfFilesUpgradeAction.java

+      ReconNamespaceSummaryManager nsSummaryManager = injector.getInstance(ReconNamespaceSummaryManager.class);
+      ReconOMMetadataManager omMetadataManager = injector.getInstance(ReconOMMetadataManager.class);
+      LOG.info("Starting full rebuild of NSSummary for REPLICATED_SIZE_OF_FILES upgrade...");
+      nsSummaryManager.rebuildNSSummaryTree(omMetadataManager);


How about if we trigger rebuild as an async operation because we have plans to show the status update of recon tasks on Overview page if they are in progress. This may ease upgrade process efficient and quicker...

tanvipenumudy · 2025-07-18T07:44:55Z

we trigger rebuild as an async operation because we have plans to show the status update of recon tasks on Overview page if they are in progress.

Thank you @devmadhuu for the review. If we look at the parentID addition in NSSummary (PR: #6492), the rebuild was triggered at a different stage and in this case, the flow for rebuild via the upgrade handler is completely different - so as per our offline discussion, HDDS-13443 should be a good starting point for providing a single point of control for triggering the NSSummary tree rebuilds in Recon.

This reverts commit 367ac09.

ArafatKhan2198

LGTM +1
Thanks a lot for working on this @tanvipenumudy.

ChenSammi · 2025-07-21T08:24:56Z

Hi @ArafatKhan2198 , @devmadhuu do you have any idea if "nsSummaryManager.rebuildNSSummaryTree(omMetadataManager)" blocks hours during Recon update, will there be any impact to Recon's other part functionality?

devmadhuu · 2025-07-22T04:59:31Z

Hi @ArafatKhan2198 , @devmadhuu do you have any idea if "nsSummaryManager.rebuildNSSummaryTree(omMetadataManager)" blocks hours during Recon update, will there be any impact to Recon's other part functionality?

@ChenSammi nsSummaryManager.rebuildNSSummaryTree(omMetadataManager) call may take sometime. It depends on the size of OM metadata. But we have tested with few varying size of DBs (ranging from 80G till 148 GB). It may take from ~2 to 4 hours. But this was when we had ACL bloat issue. So if keeping that aside, it can come down from ~20 mins to 1 hour.

...on/src/test/java/org/apache/hadoop/ozone/recon/TestReconInsightsForDeletedDirectoriesEC.java

ArafatKhan2198 · 2025-07-22T08:25:04Z

Hi @ArafatKhan2198 , @devmadhuu do you have any idea if "nsSummaryManager.rebuildNSSummaryTree(omMetadataManager)" blocks hours during Recon update, will there be any impact to Recon's other part functionality?

@ChenSammi nsSummaryManager.rebuildNSSummaryTree(omMetadataManager) call may take sometime. It depends on the size of OM metadata. But we have tested with few varying size of DBs (ranging from 80G till 148 GB). It may take from ~2 to 4 hours. But this was when we had ACL bloat issue. So if keeping that aside, it can come down from ~20 mins to 1 hour.

Quick Facts about the DB I recently tested at :

Total Directories: 168 million
Total Keys: 237 million
Time to build the tree: Reprocessing for NSSummaryTaskWithFSO completed in 11,957,113 milliseconds (~3 hours, 19 minutes, 17 seconds).

cc: @ChenSammi @tanvipenumudy

devabhishekpal

Thanks @tanvipenumudy.
This LGTM, +1

tanvipenumudy · 2025-07-22T14:53:50Z

Thank you @devabhishekpal, @ChenSammi, @devmadhuu, @ArafatKhan2198, @adoroszlai and @sumitagrawl for reviewing the patch!

tanvipenumudy requested review from ArafatKhan2198 and sumitagrawl June 5, 2025 17:01

tanvipenumudy marked this pull request as draft June 5, 2025 18:03

tanvipenumudy marked this pull request as ready for review June 5, 2025 19:20

ivandika3 added the recon label Jun 6, 2025

devabhishekpal reviewed Jun 9, 2025

View reviewed changes

devabhishekpal reviewed Jun 23, 2025

View reviewed changes

hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/OMDBInsightEndpoint.java Show resolved Hide resolved

devabhishekpal reviewed Jun 23, 2025

View reviewed changes

hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/OMDBInsightEndpoint.java Outdated Show resolved Hide resolved

devabhishekpal reviewed Jun 23, 2025

View reviewed changes

tanvipenumudy marked this pull request as draft June 25, 2025 08:56

tanvipenumudy changed the base branch from master to HDDS-13177 June 25, 2025 08:59

tanvipenumudy mentioned this pull request Jun 25, 2025

HDDS-13180. Add replicatedSizeOfFiles to NSSummary #8698

Closed

tanvipenumudy marked this pull request as ready for review July 7, 2025 07:19

tanvipenumudy marked this pull request as draft July 7, 2025 08:45

tanvipenumudy force-pushed the HDDS-13180 branch from 6bce18a to 7e12a08 Compare July 8, 2025 10:18

tanvipenumudy marked this pull request as ready for review July 8, 2025 10:19

tanvipenumudy requested a review from ChenSammi July 9, 2025 04:57

sumitagrawl approved these changes Jul 11, 2025

View reviewed changes

ChenSammi reviewed Jul 14, 2025

View reviewed changes

hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/OMDBInsightEndpoint.java Outdated Show resolved Hide resolved

ChenSammi reviewed Jul 14, 2025

View reviewed changes

hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/fs/ozone/TestLeaseRecovery.java Outdated Show resolved Hide resolved

tanvipenumudy mentioned this pull request Jul 14, 2025

HDDS-13188. Track total pending deletion size for FSO directories in OM via Recon #8591

Merged

tanvipenumudy marked this pull request as draft July 16, 2025 07:57

Penumudy Tanvi added 5 commits July 16, 2025 13:43

HDDS-13180. Add replicatedSizeOfFiles to NSSummary

897dd31

Reduce duplication in TestOmDBInsightEndPoint#getOmKeyInfo

853ebc5

Fix PMD failures

960ced2

Add integration test to verify for EC ReplicationConfig

d90890a

Fix checkstyle

74c2075

Penumudy Tanvi added 4 commits July 16, 2025 13:43

Update Java doc for OMDBInsightEndpoint#fetchSizeForDeletedDirectory

22c3281

Introduce upgrade framework for ReplicatedSizeOfFilesUpgradeAction

ce31658

Fix TestNSSummaryTaskWithFSO failures

24702dc

Add new TestDataUtil.createVolumeAndBucket

dc6ae02

tanvipenumudy force-pushed the HDDS-13180 branch from a13752e to dc6ae02 Compare July 16, 2025 08:14

tanvipenumudy marked this pull request as ready for review July 16, 2025 08:15

Fix compile failures

0763ff3

devmadhuu reviewed Jul 17, 2025

View reviewed changes

Penumudy Tanvi added 3 commits July 21, 2025 11:23

Remove getters and setters in parent test

367ac09

Add more tests

db5ad70

Revert "Remove getters and setters in parent test"

4e96ae0

This reverts commit 367ac09.

ArafatKhan2198 approved these changes Jul 21, 2025

View reviewed changes

adoroszlai reviewed Jul 22, 2025

View reviewed changes

...on/src/test/java/org/apache/hadoop/ozone/recon/TestReconInsightsForDeletedDirectoriesEC.java Outdated Show resolved Hide resolved

Penumudy Tanvi added 2 commits July 22, 2025 15:33

Make TestReconInsightsForDeletedDirectories parameterized

3af9d24

Fix findbugs

138959f

devabhishekpal approved these changes Jul 22, 2025

View reviewed changes

tanvipenumudy merged commit 01cf373 into apache:HDDS-13177 Jul 22, 2025
42 checks passed

priyeshkaratha pushed a commit to priyeshkaratha/ozone that referenced this pull request Oct 7, 2025

HDDS-13180. Add replicatedSizeOfFiles to NSSummary (apache#8568)

408eb63

HDDS-13180. Add replicatedSizeOfFiles to NSSummary #8568

HDDS-13180. Add replicatedSizeOfFiles to NSSummary #8568

Uh oh!

Conversation

tanvipenumudy commented Jun 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

What is the link to the Apache JIRA

How was this patch tested?

Uh oh!

devabhishekpal left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

devabhishekpal left a comment

Choose a reason for hiding this comment

Uh oh!

tanvipenumudy commented Jul 9, 2025

Uh oh!

devabhishekpal commented Jul 9, 2025

Uh oh!

sumitagrawl left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ChenSammi Jul 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tanvipenumudy Jul 14, 2025

Choose a reason for hiding this comment

Uh oh!

tanvipenumudy Jul 15, 2025

Choose a reason for hiding this comment

Uh oh!

ChenSammi Jul 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

devmadhuu Jul 17, 2025

Choose a reason for hiding this comment

Uh oh!

tanvipenumudy Jul 22, 2025

Choose a reason for hiding this comment

Uh oh!

tanvipenumudy Jul 22, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

devmadhuu left a comment

Choose a reason for hiding this comment

Uh oh!

devmadhuu Jul 17, 2025

Choose a reason for hiding this comment

Uh oh!

tanvipenumudy commented Jul 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ArafatKhan2198 left a comment

Choose a reason for hiding this comment

Uh oh!

ChenSammi commented Jul 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

devmadhuu commented Jul 22, 2025

Uh oh!

Uh oh!

ArafatKhan2198 commented Jul 22, 2025

Uh oh!

devabhishekpal left a comment

Choose a reason for hiding this comment

Uh oh!

tanvipenumudy commented Jul 22, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

tanvipenumudy commented Jun 5, 2025 •

edited

Loading

ChenSammi Jul 14, 2025 •

edited

Loading

ChenSammi Jul 17, 2025 •

edited

Loading

tanvipenumudy commented Jul 18, 2025 •

edited

Loading

ChenSammi commented Jul 21, 2025 •

edited

Loading