Hdds 13184 persist block size

[priyeshkaratha]

What changes were proposed in this pull request?

Provide a one-liner summary of the changes in the PR Title field above.
It should be in the form of HDDS-1234. Short summary of the change.

Please describe your PR in detail:

• What changes are proposed in the PR? and Why? It would be better if it is written from third person's
  perspective not just for the reviewer.
• Provide as much context and rationale for the pull request as possible. It could be copy-paste from
  the Jira's description if the jira is well defined.
• If it is complex code, describe the approach used to solve the issue. If possible attach design doc,
  issue investigation, github discussion, etc.

Examples of well-written pull requests:

• #3980
• #5265
• #4701
• #5283
• #5300

What is the link to the Apache JIRA

Please create an issue in ASF JIRA before opening a pull request, and you need to set the title of the pull
request which starts with the corresponding JIRA issue number. (e.g. HDDS-XXXX. Fix a typo in YYY.)

(Please replace this section with the link to the Apache JIRA)

How was this patch tested?

(Please explain how this patch was tested. Ex: unit tests, manual tests, workflow run on the fork git repo.)
(If this patch involves UI changes, please attach a screenshot; otherwise, remove this.)

[gemini-code-assist]

Summary of Changes

Hello @priyeshkaratha, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances the block deletion mechanism within HDDS by introducing the capability to persist and summarize the size information of deleted blocks. This change moves beyond merely tracking block IDs to providing a comprehensive view of disk space utilization for blocks awaiting physical deletion. It involves updates to core data structures, RPC protocols, and state management to ensure data consistency and enable better resource management and monitoring.

Highlights

• Deleted Block Size Persistence: The system now persists the actual size and replicated size of deleted blocks, rather than just their IDs. This is crucial for accurate quota management and resource tracking.
• New Layout Feature: A new layout feature, STORAGE_DATA_DISTRIBUTION, has been introduced to enable and manage the persistence of deleted block size information.
• Deleted Blocks Transaction Summary: A new DeletedBlocksTransactionSummary protobuf message and corresponding logic have been added to aggregate and provide a high-level overview of all pending deleted block transactions, including total count, total block size, and total replicated size.
• Client and CLI Access: The DeletedBlocksTransactionSummary is now accessible via new methods in the ScmClient and StorageContainerLocationProtocol interfaces, as well as through a new CLI command (ozone scm deletedBlocksTxn summary).
• Atomic DB Updates: The SCM HA DB transaction buffer now supports pausing and resuming auto-flush, ensuring that updates to deleted block transactions and their summary are atomic, maintaining data consistency.
• Metrics Enhancement: New metrics have been added to track the efficiency of fetching deleted block size information, distinguishing between data retrieved from in-memory cache versus the database.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces a significant feature to persist block size information within deleted block transactions. This will enable more accurate tracking of storage space reclamation. The changes are comprehensive, spanning protobuf definitions, client and server logic, metrics, and new CLI commands for administration. A new layout feature flag, STORAGE_DATA_DISTRIBUTION, is introduced to manage this change. The inclusion of extensive integration tests is a great addition and ensures the feature's robustness.

I have a couple of minor suggestions for improvement:

1. The description for the new layout feature STORAGE_DATA_DISTRIBUTION appears to be a copy-paste error and should be corrected to accurately reflect its purpose.
2. There is a small typo in a comment within DeletedBlockLogImpl.java referring to the new layout feature.

Overall, this is a well-implemented and valuable feature. The code is well-structured and the changes are consistent across the codebase.