HDDS-13026. KeyDeletingService should also delete RenameEntries #8447

swamirishi · 2025-05-14T00:48:34Z

What changes were proposed in this pull request?

Currently the entry from rename entries are never removed from snapshot renamed table which can lead to a bloat in the table size. Key deleting service should use ReclaimableRenameEntry to remove entries which don't even point to any object in the previous snapshot. This can be done as part of the snapshot deep cleaning and AOS garbage collection service

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-13026

How was this patch tested?

Additional Unit tests

… each and every ratis request Change-Id: I5c29c572df9d2240b1d58fbc88826eb5ed8ad881

…shotInfo Change-Id: I16d2881af973799773335343debd856cb763f72b

Change-Id: I67d685aff12daaf98cf6a8eb5f6a5750c4cf6394

Change-Id: I4215cacc0f0486a7aa46b60971149483028bed38

Change-Id: I7a05f30d0a58538708e5acdbce36d836b6a5542e

Change-Id: I5a9e9dd3cc74e185b1f0745af7f2fee1026d125a # Conflicts: # hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/service/AbstractKeyDeletingService.java # hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/service/SnapshotDeletingService.java

Change-Id: I86c0adae571debf1e35e86660beb8c6a62d8b6fc

Change-Id: I403f238e8fd1178540da893f73a42aa2223572ad

Change-Id: Id3202dbe7a5a71c43526e80af34b18dd7b2ed66e

Change-Id: I17c1771c7630292fded0eebfbe6ee14f97798506

Change-Id: I03e67781351ed3a18666b32a12109141a5a4a34d

Change-Id: I1712f2dd4138a1f679eb4e4676b3d5ac41c80583

Change-Id: Id4a53d3eb8f23031ef517b3793dc73b067b11d62

Change-Id: I99896a697c523d6c174a06e66935a4be3b2bd541

jojochuang

huge code piece. need time to review.

jojochuang · 2025-05-16T21:34:18Z

...anager/src/main/java/org/apache/hadoop/ozone/om/request/snapshot/OMSnapshotPurgeRequest.java

      // current snapshot is deleted. We can potentially
      // reclaim more keys in the next snapshot.
      snapInfo.setDeepClean(false);
+      snapInfo.setDeepCleanedDeletedDir(false);


this is part of #8451?

jojochuang · 2025-05-16T23:45:03Z

...ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/service/KeyDeletingService.java

+    /**
+     *
+     * @param currentSnapshotInfo if null, deleted directories in AOS should be processed.
+     * @param keyManager KeyManager of the underlying store.
+     */


/** * Scans the OM database for keys or rename entries that are eligible for deletion * in the given store (either the active store or a specific snapshot). * * It collects a batch of eligible keys and/or renamed entries up to the specified * remainNum limit, submits deletion requests to SCM, and, on successful confirmation, * removes the entries from the metadata store. The method also updates snapshot * properties related to reclaimed space and deep cleaning flags as required. * * @param currentSnapshotInfo If non-null, processes deleted keys for a specific snapshot; * if null, processes for the active object store. * @param keyManager The KeyManager instance for accessing keys in the underlying store. * @param remainNum The maximum number of entries to process in this invocation. * @return The number of entries remaining to be processed after this method runs. * @throws IOException If any error occurs during OM DB or SCM operations. */

jojochuang · 2025-05-16T23:47:19Z

...ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/service/KeyDeletingService.java

+     * @param currentSnapshotInfo if null, deleted directories in AOS should be processed.
+     * @param keyManager KeyManager of the underlying store.
+     */
+    private int processDeletedKeysForStore(SnapshotInfo currentSnapshotInfo, KeyManager keyManager,


this method appears to be extracted out of the below call(). But why is it much longer. Can it be broken up into smaller pieces.

hadoop-ozone/interface-client/src/main/proto/OmClientProtocol.proto

...one-manager/src/main/java/org/apache/hadoop/ozone/om/service/AbstractKeyDeletingService.java

jojochuang · 2025-05-16T23:57:20Z

also is there a test to verify that rename entries are removed after key deletion?

Change-Id: I76d096c4176bbf5508b6f75160c3524c1a04c5f0 # Conflicts: # hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/lock/OzoneManagerLock.java # hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/snapshot/TestSnapshotDirectoryCleaningService.java # hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/service/AbstractKeyDeletingService.java # hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/service/KeyDeletingService.java # hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/service/TestKeyDeletingService.java

Change-Id: Ibfcfe5d817ac6ac83f76abe3af08b5794bbb4b3e

Change-Id: I8137e7f06014951719dbf8736124f261193f2143 # Conflicts: # hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/KeyManager.java # hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/KeyManagerImpl.java # hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/service/AbstractKeyDeletingService.java # hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/service/KeyDeletingService.java # hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/service/TestKeyDeletingService.java

Change-Id: I7af63cb80766fc83ec75ef9f7ad0ca8d299caa7a

Change-Id: Ib4c0199f50e5efb8f40b4e5c897d4b33acdf9d62

Change-Id: I67ca2fea367627a7a0ad97312ae450c74fabe95b

swamirishi · 2025-05-24T03:13:21Z

huge code piece. need time to review.

@jojochuang this pr size should have dropped significantly given that #8450 is merged

Change-Id: Ie9efcc5f15cee9d58cfcea7e373882c792cdd96b

Change-Id: I0f7cafce55e87884a0eb9fde3c5e15004b4cdb71

swamirishi · 2025-05-24T04:58:10Z

also is there a test to verify that rename entries are removed after key deletion?

Done adding a test case.

jojochuang

Looks good to me.

Optional, but we could consider adding a metric for removed renamed table entries. metadataManager.countRowsInTable(snapshotRenamedTable) should only be used in tests not used in production metrics.

.../ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/key/OMKeyPurgeResponse.java

Change-Id: I046604c6d5e7c049c99f6e65010a1a40dfbda965

Change-Id: Ifd2c97c3d932fb1335eed97b4535c0b6256add4b

...ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/OMRequestTestUtils.java

Change-Id: I16f7492a085ae56d6ce6402ee5fa2e102636d4bf

swamirishi · 2025-05-28T00:35:32Z

Thank you for reviewing the patch @jojochuang & @smengcl

smengcl · 2025-05-28T18:07:08Z

Thanks @swamirishi for the patch.

HDDS-13104. Move auditparser acceptance tests under debug HDDS-12714. Create acceptance test framework for debug and repair tools (apache#8510) HDDS-13026. KeyDeletingService should also delete RenameEntries (apache#8447) HDDS-12928. datanode min free space configuration (apache#8388) Move the test under test-debug-tools.sh Export om Kinit om Set hostname to om1 Export om and not om1 Remove unsecured code

# This is the 1st commit message: HDDS-13104. Move auditparser acceptance tests under debug # This is the commit message #2: HDDS-12714. Create acceptance test framework for debug and repair tools (apache#8510) # This is the commit message #3: HDDS-13026. KeyDeletingService should also delete RenameEntries (apache#8447) # This is the commit message #4: HDDS-12928. datanode min free space configuration (apache#8388) # This is the commit message #5: Move the test under test-debug-tools.sh # This is the commit message #6: Export om # This is the commit message #7: Kinit om # This is the commit message #8: Set hostname to om1

…239-container-reconciliation Commits: 80 commits 5e273a4 HDDS-12977. Fail build on dependency problems (apache#8574) 5081ba2 HDDS-13034. Refactor DirectoryDeletingService to use ReclaimableDirFilter and ReclaimableKeyFilter (apache#8546) e936e4d HDDS-12134. Implement Snapshot Cache lock for OM Bootstrap (apache#8474) 31d13de HDDS-13165. [Docs] Python client developer guide. (apache#8556) 9e6955e HDDS-13205. Bump common-custom-user-data-maven-extension to 2.0.3 (apache#8581) 750b629 HDDS-13203. Bump Bouncy Castle to 1.81 (apache#8580) ba5177e HDDS-13202. Bump build-helper-maven-plugin to 3.6.1 (apache#8579) 07ee5dd HDDS-13204. Bump awssdk to 2.31.59 (apache#8582) e1964f2 HDDS-13201. Bump jersey2 to 2.47 (apache#8578) 81295a5 HDDS-13013. [Snapshot] Add metrics and tests for snapshot operations. (apache#8436) b3d75ab HDDS-12976. Clean up unused dependencies (apache#8521) e0f08b2 HDDS-13179. rename-generated-config fails on re-compile without clean (apache#8569) f388317 HDDS-12554. Support callback on completed reconfiguration (apache#8391) c13a3fe HDDS-13154 Link more Grafana dashboard json files to the Observability user doc (apache#8533) 2a761f7 HDDS-11967. [Docs]DistCP Integration in Kerberized environment. (apache#8531) 81fc4c4 HDDS-12550. Use DatanodeID instead of UUID in NodeManager CommandQueue. (apache#8560) 2360af4 HDDS-13169. Intermittent failure in testSnapshotOperationsNotBlockedDuringCompaction (apache#8553) f19789d HDDS-13170. Reclaimable filter should always reclaim entries when buckets and volumes have already been deleted (apache#8551) 315ef20 HDDS-13175. Leftover reference to OM-specific trash implementation (apache#8563) 902e715 HDDS-13159. Refactor KeyManagerImpl for getting deleted subdirectories and deleted subFiles (apache#8538) 46a93d0 HDDS-12817. Addendum rename ecIndex to replicaIndex in chunkinfo output (apache#8552) 19b9b9c HDDS-13166. Set pipeline ID in BlockExistenceVerifier to avoid cached pipeline with different node (apache#8549) b3ff67c HDDS-13068. Validate Container Balancer move timeout and replication timeout configs (apache#8490) 7a7b9a8 HDDS-13139. Introduce bucket layout flag in freon rk command (apache#8539) 3c25e7d HDDS-12595. Add verifier for container replica states (apache#8422) 6d59220 HDDS-13104. Move auditparser acceptance test under debug (apache#8527) 8e8c432 HDDS-13071. Documentation for Container Replica Debugger Tool (apache#8485) 0e8c8d4 HDDS-13158. Bump junit to 5.13.0 (apache#8537) 8e552b4 HDDS-13157. Bump exec-maven-plugin to 3.5.1 (apache#8534) 168f690 HDDS-13155. Bump jline to 3.30.4 (apache#8535) cc1e4d1 HDDS-13156. Bump awssdk to 2.31.54 (apache#8536) 3bfb7af HDDS-13136. KeyDeleting Service should not run for already deep cleaned snapshots (apache#8525) 006e691 HDDS-12503. Compact snapshot DB before evicting a snapshot out of cache (apache#8141) 568b228 HDDS-13067. Container Balancer delete commands should not be sent with an expiration time in the past (apache#8491) 53673c5 HDDS-11244. OmPurgeDirectoriesRequest should clean up File and Directory tables of AOS for deleted snapshot directories (apache#8509) 07f4868 HDDS-13099. ozone admin datanode list ignores --json flag when --id filter is used (apache#8500) 08c0ab8 HDDS-13075. Fix default value in description of container placement policy configs (apache#8511) 58c87a8 HDDS-12177. Set runtime scope where missing (apache#8513) 10c470d HDDS-12817. Add EC block index in the ozone debug replicas chunk-info (apache#8515) 7027ab7 HDDS-13124. Respect config hdds.datanode.use.datanode.hostname when reading from datanode (apache#8518) b8b226c HDDS-12928. datanode min free space configuration (apache#8388) fd3d70c HDDS-13026. KeyDeletingService should also delete RenameEntries (apache#8447) 4c1c6cf HDDS-12714. Create acceptance test framework for debug and repair tools (apache#8510) fff80fc HDDS-13118. Remove duplicate mockito-core dependency from hdds-test-utils (apache#8508) 10d5555 HDDS-13115. Bump awssdk to 2.31.50 (apache#8505) 360d139 HDDS-13017. Fix warnings due to non-test scoped test dependencies (apache#8479) 1db1cca HDDS-13116. Bump jline to 3.30.3 (apache#8504) 322ca93 HDDS-13025. Refactor KeyDeletingService to use ReclaimableKeyFilter (apache#8450) 988b447 HDDS-5287. Document S3 ACL classes (apache#8501) 64bb29d HDDS-12777. Use module-specific name for generated config files (apache#8475) 54ed115 HDDS-9210. Update snapshot chain restore test to incorporate snapshot delete. (apache#8484) 87dfa5a HDDS-13014. Improve PrometheusMetricsSink#normalizeName performance (apache#8438) 7cdc865 HDDS-13100. ozone admin datanode list --json should output a newline at the end (apache#8499) 9cc4194 HDDS-13089. [snapshot] Add an integration test to verify snapshotted data can be read by S3 SDK client (apache#8495) cb9867b HDDS-13065. Refactor SnapshotCache to return AutoCloseSupplier instead of ReferenceCounted (apache#8473) a88ff71 HDDS-10979. Support STANDARD_IA S3 storage class to accept EC replication config (apache#8399) 6ec8f85 HDDS-13080. Improve delete metrics to show number of timeout DN command from SCM (apache#8497) 3bb8858 HDDS-12378. Change default hdds.scm.safemode.min.datanode to 3 (apache#8331) 0171bef HDDS-13073. Set pipeline ID in checksums verifier to avoid cached pipeline with different node (apache#8480) 5c7726a HDDS-11539. OzoneClientCache `@PreDestroy` is never called (apache#8493) a8ed19b HDDS-13031. Implement a Flat Lock resource in OzoneManagerLock (apache#8446) e9e8b30 HDDS-12935. Support unsigned chunked upload with STREAMING-UNSIGNED-PAYLOAD-TRAILER (apache#8366) 7590268 HDDS-13079. Improve logging in DN for delete operation. (apache#8489) 435fe7e HDDS-12870. Fix listObjects corner cases (apache#8307) eb5dabd HDDS-12926. Remove *.tmp.* exclusion in DU (apache#8486) eeb98c7 HDDS-13030. Snapshot Purge should unset deep cleaning flag for next 2 snapshots in the chain (apache#8451) 6bf121e HDDS-13032. Support proper S3OwnerId representation (apache#8478) 5d1b43d HDDS-13076. Refactor OzoneManagerLock class to rename Resource class to LeveledResource (apache#8482) bafe6d9 HDDS-13064. [snapshot] Add test coverage for SnapshotUtils.isBlockLocationInfoSame() (apache#8476) 7035846 HDDS-13040. Add user doc highlighting the difference between Ozone ACL and S3 ACL. (apache#8457) 1825cdf HDDS-13049. Deprecate VolumeName & BucketName in OmKeyPurgeRequest and prevent Key version purge on Block Deletion Failure (apache#8463) 211c76c HDDS-13060. Change NodeManager.addDatanodeCommand(..) to use DatanodeID (apache#8471) f410238 HDDS-13061. Add test for key ACL operations without permission (apache#8472) d1a2f48 HDDS-13057. Increment block delete processed transaction counts regardless of log level (apache#8466) 0cc6fcc HDDS-13043. Replace != with assertNotEquals in TestSCMContainerPlacementRackAware (apache#8470) e1c779a HDDS-13051. Use DatanodeID in server-scm. (apache#8465) 35e1126 HDDS-13042. [snapshot] Add future proofing test cases for unsupported file system API (apache#8458) 619c05d HDDS-13008. Exclude same SST files when calculating full snapdiff (apache#8423) 21b49d3 HDDS-12965. Fix warnings about "used undeclared" dependencies (apache#8468) 8136119 HDDS-13048. Create new module for Recon integration tests (apache#8464) Conflicts: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/NodeManager.java

…ntries (apache#8447) (cherry picked from commit fd3d70c) Conflicts: hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/snapshot/TestSnapshotDeletingServiceIntegrationTest.java hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/DeletingServiceMetrics.java hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/KeyManager.java hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/key/TestOMKeyPurgeRequestAndResponse.java hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/service/TestKeyDeletingService.java Change-Id: I635564b6f5ad52cd1f0c454eeeb6f9a75c8493ca

swamirishi added 14 commits May 11, 2025 20:12

HDDS-13021. AbstractKeyDeletingService should have unique call id for…

8c626c6

… each and every ratis request Change-Id: I5c29c572df9d2240b1d58fbc88826eb5ed8ad881

HDDS-13022. Segragate Exclusive Dir Size and ExclusiveKeySize in Snap…

50b1a23

…shotInfo Change-Id: I16d2881af973799773335343debd856cb763f72b

HDDS-13022. Add Test OzoneSnapshot client code

685aac2

Change-Id: I67d685aff12daaf98cf6a8eb5f6a5750c4cf6394

HDDS-13022. Add hashcode for OzoneSnapshots

c29d3d8

Change-Id: I4215cacc0f0486a7aa46b60971149483028bed38

HDDS-13022. Add hashcode for OzoneSnapshots

49184f6

Change-Id: I7a05f30d0a58538708e5acdbce36d836b6a5542e

HDDS-13022. Address review comments

ea94417

Change-Id: I86c0adae571debf1e35e86660beb8c6a62d8b6fc

HDDS-13022. Address review comments

7bcbda6

Change-Id: I403f238e8fd1178540da893f73a42aa2223572ad

HDDS-13031. Remove Lock set bits for SNAPSHOT_GC_LOCK

b86ec7e

Change-Id: Id3202dbe7a5a71c43526e80af34b18dd7b2ed66e

HDDS-13026. KeyDeleting service should also delete RenameEntries

617cf00

Change-Id: I17c1771c7630292fded0eebfbe6ee14f97798506

Merge remote-tracking branch 'origin/HDDS-13031' into HEAD

33435cb

Change-Id: I03e67781351ed3a18666b32a12109141a5a4a34d

HDDS-13031. Fix Locking setbit issue

40b6a83

Change-Id: I1712f2dd4138a1f679eb4e4676b3d5ac41c80583

HDDS-13031. Fix checkstyle

c962507

Change-Id: Id4a53d3eb8f23031ef517b3793dc73b067b11d62

Merge remote-tracking branch 'origin/HDDS-13031' into HEAD

29b1c6d

Change-Id: I99896a697c523d6c174a06e66935a4be3b2bd541

swamirishi requested review from jojochuang and smengcl May 14, 2025 09:50

swamirishi added the snapshot https://issues.apache.org/jira/browse/HDDS-6517 label May 14, 2025

swamirishi requested a review from sadanand48 May 15, 2025 14:43

jojochuang reviewed May 16, 2025

View reviewed changes

swamirishi added 6 commits May 20, 2025 18:37

HDDS-13026. Fix checkstyle

4b84307

Change-Id: Ibfcfe5d817ac6ac83f76abe3af08b5794bbb4b3e

HDDS-13026. Merge with master

6116cba

Change-Id: I7af63cb80766fc83ec75ef9f7ad0ca8d299caa7a

HDDS-13026. Merge with master

3af0ac6

Change-Id: Ib4c0199f50e5efb8f40b4e5c897d4b33acdf9d62

HDDS-13026. Add OmKeyPurgeRequest changes

6249698

Change-Id: I67ca2fea367627a7a0ad97312ae450c74fabe95b

swamirishi added 2 commits May 24, 2025 00:03

HDDS-13026. Add test case

84dcb6b

Change-Id: Ie9efcc5f15cee9d58cfcea7e373882c792cdd96b

HDDS-13026. Add test case

ea8a3bc

Change-Id: I0f7cafce55e87884a0eb9fde3c5e15004b4cdb71

jojochuang reviewed May 26, 2025

View reviewed changes

.../ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/key/OMKeyPurgeResponse.java Show resolved Hide resolved

swamirishi added 2 commits May 26, 2025 19:31

HDDS-13026. Add rename entries purged in metrics

fba3c21

Change-Id: I046604c6d5e7c049c99f6e65010a1a40dfbda965

HDDS-13026. Fix test

4f89ca6

Change-Id: Ifd2c97c3d932fb1335eed97b4535c0b6256add4b

swamirishi marked this pull request as ready for review May 27, 2025 17:43

jojochuang approved these changes May 27, 2025

View reviewed changes

smengcl changed the title ~~HDDS-13026. KeyDeleting service should also delete RenameEntries~~ HDDS-13026. KeyDeletingService should also delete RenameEntries May 27, 2025

smengcl reviewed May 27, 2025

View reviewed changes

...ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/OMRequestTestUtils.java Outdated Show resolved Hide resolved

HDDS-13026. Fix alignment

7786270

Change-Id: I16f7492a085ae56d6ce6402ee5fa2e102636d4bf

swamirishi merged commit fd3d70c into apache:master May 28, 2025
42 checks passed

HDDS-13026. KeyDeletingService should also delete RenameEntries #8447

HDDS-13026. KeyDeletingService should also delete RenameEntries #8447

Uh oh!

Conversation

swamirishi commented May 14, 2025

What changes were proposed in this pull request?

What is the link to the Apache JIRA

How was this patch tested?

Uh oh!

jojochuang left a comment

Choose a reason for hiding this comment

Uh oh!

jojochuang May 16, 2025

Choose a reason for hiding this comment

Uh oh!

jojochuang May 16, 2025

Choose a reason for hiding this comment

Uh oh!

jojochuang May 16, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

jojochuang commented May 16, 2025

Uh oh!

swamirishi commented May 24, 2025

Uh oh!

swamirishi commented May 24, 2025

Uh oh!

jojochuang left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

swamirishi commented May 28, 2025

Uh oh!

Uh oh!

smengcl commented May 28, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants