HDDS-13432. Accelerating Namespace Usage Calculation in Recon using - Aggregation Approach

[ArafatKhan2198]

What changes were proposed in this pull request?

This pull request introduces a "materialized view" approach for Recon's Namespace Summary (NSSummary), significantly improving the speed of namespace usage calculations.

Approach and Implementation Details:

1. Materialized Totals in NSSummary:
   • The numOfFiles and sizeOfFiles fields in org.apache.hadoop.ozone.recon.api.types.NSSummary now store the total number and size of files within a directory and all its subdirectories, not just direct files. This enables constant-time (O(1)) queries for aggregate usage.
2. Real-time Propagation on Events:
   • The org.apache.hadoop.ozone.recon.tasks.NSSummaryTaskDbEventHandler is enhanced to immediately propagate changes:
     • File Events (PUT/DELETE): When a file is added or deleted, its size and count changes are propagated upwards through its entire parent chain to update ancestor NSSummary totals.
     • Directory Deletion Events: When a directory is deleted, its NSSummary is unlinked (its parentId set to 0L), and its total file count and size (at the moment of deletion) are decremented from its parent and all ancestors. The deleted directory's NSSummary itself retains its historical totals.
3. Idempotent Deletion Handling:
   • The logic ensures that deletion events are handled idempotently. If a directory is deleted first (and its totals propagated), subsequent deletion events for files within that already unlinked directory will not cause redundant decrements in ancestor totals. Similarly, if a file is deleted first, the subsequent deletion of its parent directory will not re-decrement already adjusted totals.
4. Simplified API Queries:
   • The org.apache.hadoop.ozone.recon.api.handlers.EntityHandler now directly uses the pre-calculated numOfFiles and sizeOfFiles from NSSummary for total counts and sizes, eliminating the need for recursive tree traversals at query time.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-13432

How was this patch tested?

Manually verified the changes, also added Unit Tests to verify the changes.

[devmadhuu]

@ArafatKhan2198 Thanks for the patch. Pls check few comments.

[devmadhuu]

especially for deletion events.

Is this true. I believe this is true even for diskUsage(Namespace usage) page as well.

[devmadhuu]

Alos pls resolve conflicts.

[ArafatKhan2198]

@devmadhuu @sumitagrawl

I found a bug in my implementation code that handles directory deletions. When a directory is deleted, the system correctly updates the parent directory's total file count and size, but it fails to accurately update the file size distribution histogram. This leads to incorrect information about the size categories of files in the parent directory.

Simple Example

Imagine a directory /home with two subdirectories:

• /home/dir_A (contains one 100-byte file)
• /home/dir_B (contains one 500-byte file)

The file size distribution for /home should show one file in the "100-byte" bin and one file in the "500-byte" bin.

When /home/dir_B is deleted, the bug causes the system to only decrement the total file count and size of /home. The file size distribution for /home would still incorrectly show a file in both the "100-byte" and "500-byte" bins, even though the 500-byte file is gone.

I have applied a fix to ensures that when a directory is deleted, the parent directory's file size distribution is correctly updated by subtracting the specific file size distribution of the deleted directory. After the fix, deleting /home/dir_B would result in /home's distribution correctly showing only one file in the "100-byte" bin.

I've thoroughly tested this fix with unit tests for both file and directory deletion scenarios to ensure everything works correctly.

[ArafatKhan2198]

Hi @sumitagrawl and @devmadhuu,

I have reverted the changes. We are no longer focusing on the File Bucket Distribution for now. We can consider implementing and tracking it later through this HDDS-13539 after further discussions. For the time being, please review the current state so we can proceed with the implementation.

Materialized Size/Count Optimization - Event Handling Rules

Every change must update the immediate parent AND propagate to ALL ancestors up to the root. This ensures every directory knows its total size/count including all subdirectories.

File Addition

What happens: New file created

Action:

1. Add file size and count to immediate parent directory
2. Propagate these additions up to ALL ancestor directories
3. Update file size distribution bucket for immediate parent only

---

File Deletion

What happens: File removed

Action:

1. Subtract file size and count from immediate parent directory
2. Propagate these subtractions up to ALL ancestor directories
3. Update file size distribution bucket for immediate parent only

---

Directory Addition

What happens: Directory explicitly created (might already exist with content)

Action:

1. Set directory name and parent relationship
2. Add directory to parent's child list
3. CRITICAL: If directory already had files (from earlier file operations), propagate those existing totals up to ALL ancestors

---

Directory Deletion

What happens: Directory and all contents removed

Action:

1. Get the directory's total size and file count (represents everything inside it)
2. Subtract these totals from immediate parent
3. Propagate these subtractions up to ALL ancestor directories
4. Remove directory from parent's child list and unlink it

[sumitagrawl]

@ArafatKhan2198 Thanks for working over this, given few comments

[sumitagrawl]

@ArafatKhan2198 Thanks for working over this, given few comments

[sumitagrawl]

LGTM

[devmadhuu]

Thanks @ArafatKhan2198 for the improved changes. LGTM now +1

[ArafatKhan2198]

Fork CI Green :- https://github.com/ArafatKhan2198/ozone/actions/runs/16797786720