HDDS-13184. Persist Block Size in Delete Transaction for SCM by priyeshkaratha · Pull Request #9272 · apache/ozone

priyeshkaratha · 2025-11-10T04:33:01Z

What changes were proposed in this pull request?

Add a DeletedBlocksTransactionSummary to count the block deletion transaction count, all involved block count, all size and replicated size of all involved block.
When block deletion transaction is persisted into SCM DB, transaction info is updated to increase in DeletedBlocksTransactionSummary. When block deletion transaction is removed from SCM DB, transaction info is updated to decrease in DeletedBlocksTransactionSummary. Any retry command of transaction doesn't affect the DeletedBlocksTransactionSummary.
Upgrading, for an existing Ozone cluster, all existing block deletion transaction prior to STORAGE_SPACE_DISTRIBUTION finalized, will be ignored to update DeletedBlocksTransactionSummary when it's removed from SCM DB. DeletedBlocksTransactionSummary only counts the transaction after STORAGE_SPACE_DISTRIBUTION is finalized.
Add a new CLI to query DeletedBlocksTransactionSummary.
Expose the DeletedBlocksTransactionSummary through metrics.
DB size increase consideration, refer to TestDeletedBlockLog#testTransactionSerializedSize comments
Performance consideration, refer to TestDeletedBlockLog#testAddRemoveTransactionPerformance comments

What is the link to the Apache JIRA

HDDS-13184

How was this patch tested?

New unit tests added. And its tested in feature branch part of #8845

…8845)

hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/upgrade/HDDSLayoutFeature.java

hadoop-hdds/interface-server/src/main/proto/ScmServerDatanodeHeartbeatProtocol.proto

...rver-scm/src/main/java/org/apache/hadoop/hdds/scm/block/DeletedBlockLogStateManagerImpl.java

...p-ozone/cli-admin/src/main/java/org/apache/hadoop/hdds/scm/cli/ContainerOperationClient.java

...rver-scm/src/main/java/org/apache/hadoop/hdds/scm/block/DeletedBlockLogStateManagerImpl.java

sumitagrawl

@priyeshkaratha given few comments

hadoop-hdds/interface-client/src/main/proto/hdds.proto

hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/block/DeletedBlockLogImpl.java

...rver-scm/src/main/java/org/apache/hadoop/hdds/scm/block/DeletedBlockLogStateManagerImpl.java

hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/block/DeletedBlockLogImpl.java

.../src/main/java/org/apache/hadoop/hdds/scm/block/SCMDeletedBlockTransactionStatusManager.java

ChenSammi

Thanks @priyeshkaratha for the contribution , and @sumitagrawl for the review.

sumitagrawl

@priyeshkaratha Some additional minor review comments are there after previous fix, please handle with new JIRA

sumitagrawl · 2025-11-20T07:33:56Z

...dds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/ha/SCMHADBTransactionBufferImpl.java

  private final AtomicLong txFlushPending = new AtomicLong(0);
  private long lastSnapshotTimeMs = 0;
  private final ReentrantReadWriteLock rwLock = new ReentrantReadWriteLock();
+  private boolean autoFlushEnabled = true;


This flag is of no meaning now, can be removed

sumitagrawl · 2025-11-20T07:38:11Z

.../src/main/java/org/apache/hadoop/hdds/scm/block/SCMDeletedBlockTransactionStatusManager.java

+  }
+
+  private void initDataDistributionData() throws IOException {
+    if (VersionedDatanodeFeatures.isFinalized(HDDSLayoutFeature.STORAGE_SPACE_DISTRIBUTION)) {


STORAGE_SPACE_DISTRIBUTION check may not be required, it will not be available before finalize and on first startup for new version,

sumitagrawl · 2025-11-20T08:36:24Z

.../src/main/java/org/apache/hadoop/hdds/scm/block/SCMDeletedBlockTransactionStatusManager.java

+
+  @Nullable
+  public DeletedBlocksTransactionSummary getTransactionSummary() {
+    if (!VersionedDatanodeFeatures.isFinalized(HDDSLayoutFeature.STORAGE_SPACE_DISTRIBUTION)) {


STORAGE_SPACE_DISTRIBUTION here also not required, by default will be 0 if not finalized, this is redundant check.

sumitagrawl · 2025-11-20T08:38:03Z

...rver-scm/src/main/java/org/apache/hadoop/hdds/scm/block/DeletedBlockLogStateManagerImpl.java

  }

+  @Override
+  public ArrayList<DeletedBlocksTransaction> getTransactionsFromDB(ArrayList<Long> txIDs) throws IOException {


No call of this api is found, unused method to be removed

ChenSammi and others added 3 commits November 11, 2025 09:38

HDDS-13184. Persist Block Size in Delete Transaction for SCM (apache#…

6690517

…8845)

fixing errors and conflicts

9a9578e

renaming layout name to STORAGE_SPACE_DISTRIBUTION

53f34b2

priyeshkaratha force-pushed the HDDS-13184_PersistBlockSize branch from 5eef1e6 to 53f34b2 Compare November 11, 2025 04:11