Skip to content

Conversation

@swamirishi
Copy link
Contributor

What changes were proposed in this pull request?

After HDDS-13034 SnapshotDirectoryCleaningService shouldn't be used anywhere, thus remove all traces of the class and also move respective functions in AbstractDeletingService to respective background services and just keeping the common ones.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-13160

How was this patch tested?

No Tests required code refactoring.

…s and deleted subFiles

Change-Id: Ic1fc709b3963cde14c2a7fb64b687322a29e642a
Change-Id: I47b24dfc3b5afa3cefbdc85ac7b3e4a9b8c94869
…ectoryFilter & ReclaimableKeyFilter

Change-Id: Iffdda403cba914ddef2b75808cfbef1a72b9a2d3
Change-Id: I11acc3782aadf8393f731adcaa2a436dd9b534ae
…ractDeletingService

Change-Id: I11ad5f48e25a7d22676a061bf43d8d168f0ae683
@swamirishi swamirishi changed the title Hdds 13160 HDDS-13160. Remove SnapshotDirectoryCleaningService and refactor AbstractDeletingService Jun 3, 2025
swamirishi added 11 commits June 3, 2025 05:33
…kets and volumes have already been deleted

Change-Id: I16dc9d8f00686320b4e98fa5691420294a7f1e2f
Change-Id: Ie5fd1406bbb8af3a9ba76440dcba9b8d8db14691
Change-Id: I61ef68263ff88daa0e53dfb9d7d8eb62495d226b
Change-Id: I2667c6d12523f4dee7cbcf7c48c93803fe84d3d4
Change-Id: Ib517326c453a156ca9c81cb2c856238d4bc19e69

# Conflicts:
#	hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/service/DirectoryDeletingService.java
Change-Id: Iba8e6ad3fd3e9b997bcaaf1a80f1ea92ae59b6f9
Change-Id: I2e5e5a95079dbda192b5305f18e4b29b0eeb620d
Change-Id: I5ed93af3b5ae794b0cfe4671ec2a851592edcb8c
Change-Id: I587f16ede2b787cd73c7f9a11c368a372be07ed7
Change-Id: Ifda5cd552ad23086cec7163d2a7983ad1dd5f5c4
…urging deleted snapshots

Change-Id: I7f0c12c53f8838e652624bb993a6c8414ef638c7
@jojochuang jojochuang requested a review from smengcl June 3, 2025 17:22
@jojochuang jojochuang added the snapshot https://issues.apache.org/jira/browse/HDDS-6517 label Jun 3, 2025
Change-Id: I4c234b75208d96a146c7498bb6c2c188a270e1b8

# Conflicts:
#	hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/KeyManagerImpl.java
#	hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/snapshot/filter/AbstractReclaimableFilterTest.java
Change-Id: I92114f919459d2f4ca716dd0f983ae6a23db85c1
Change-Id: I87ee106c43c88cc875d8fb39ac29c7b06d797394
Change-Id: Iac3af98a7e568a135073b6704a6ad5a5fac7b427
Change-Id: Idd2afd9625c7bd811f32e4e307d18e604d5a2583
Change-Id: I9b7b41cf667e03d48120a4201757e445227924f7
Change-Id: I2ff1cf3ecf3baa00a5c5646901f6c9ffdbe6e370
Change-Id: I28997dbafe0f6eba2fc02d00310e55e7c33b0dda
Change-Id: I5ae8e13ec670d3fc639be26450f6ee21a9fd48ea
swamirishi added 17 commits June 5, 2025 21:42
Change-Id: Ibe4244ba9eac58eeb86ff4a5a65bd42b15b2a8ae
Change-Id: Ieda3456493b1720e87b7fdc8de660621bd4d26fd
Change-Id: I4bffdc63679560abed1ff614958926de3f1a4cc2
Change-Id: I2c74594424fac70e62750815b45daf3780f7d85c
Change-Id: I175d2212e70c9acc4a9fc1d8bd760e3e685ad588
Change-Id: I186d410df06fd53526c4b3d0c6c9fc8f8a5e0a97
Change-Id: I4b1da4c1291a5bce2dfb9d75a9f21f5d88f6e1cd
Change-Id: I7211fe692fb036d578535cbd06b88db634e228d3

# Conflicts:
#	hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/service/DirectoryDeletingService.java
Change-Id: I2b38ebc6bfeaba0935d568864605efc53e9eb222
Change-Id: I76b3d086daf4c2b90af2ef5df0e53542e164c52b
Change-Id: I218dfbf383800d2097df47141ee909fc45f8cfa7
Change-Id: I613d406fb84cc3707a6a4a5208d4baf901347e8a

# Conflicts:
#	hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/service/DirectoryDeletingService.java
Change-Id: I2ebcf278aaa610c84ab0da329c2837e7ef97dfa1
Change-Id: Id45068e45efa8d121dcb179cb1e286f4076b8979
Change-Id: I232f2c4915b1ed063729a4758588abc98569109d
Change-Id: If0cd16b56a5e3caeb28f3c6102d6a42c8309f548

# Conflicts:
#	hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/service/TestDirectoryDeletingServiceWithFSO.java
#	hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/service/AbstractKeyDeletingService.java
#	hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/service/DirectoryDeletingService.java
Change-Id: If3b294f9501cd8fb2c0a728e03013dcd911edc58
@swamirishi
Copy link
Contributor Author

Review by Co-Pilot:
Based on the diff you provided from the PR (https://github.com/apache/ozone/pull/8547/files), here is a thorough analysis of all functions moved from AbstractKeyDeletingService to DirectoryDeletingService and KeyDeletingService, with a line-by-line check for any changes:


1. Functions moved to DirectoryDeletingService

a. optimizeDirDeletesAndSubmitRequest

  • Moved to: DirectoryDeletingService (was previously in AbstractKeyDeletingService)
  • Diff/Change:
    The code is copied as-is. No logic change.
    • Parameters, variable names, and logic remain identical.
    • No line differences detected.
  • Conclusion: No change except location.

b. prepareDeleteDirRequest

  • Moved to: DirectoryDeletingService
  • Diff/Change:
    Copied as-is.
    • All steps (step-0 to step-3), variables, and return conditions are identical.
  • Conclusion: No change except location.

c. wrapPurgeRequest

  • Moved to: DirectoryDeletingService
  • Diff/Change:
    No logic change.
    • Still returns a built PurgePathRequest using the same parameters and logic.
  • Conclusion: No change except location.

d. submitPurgePaths

  • Moved to: DirectoryDeletingService
  • Diff/Change:
    Only two minor differences:
    • The method is now private (was protected).
    • Uses getClientId().toString() instead of direct field access clientId.toString().
    • All logic, structure, and error handling are otherwise identical.
  • Conclusion: Only access modifier and clientId access changed; logic is the same.

e. getDeletedDirsCount, getMovedDirsCount, getMovedFilesCount

  • Moved to: DirectoryDeletingService
    (Were previously in AbstractKeyDeletingService)
  • Diff/Change:
    Method bodies are unchanged—simply return values from atomic counters.
  • Conclusion: No change except location.

2. Functions moved to KeyDeletingService

a. processKeyDeletes

  • Moved to: KeyDeletingService (was previously in AbstractKeyDeletingService)
  • Diff/Change:
    Logic is copied as-is:
    • SCM block deletion, timing, logging, and call to submitPurgeKeysRequest are all identical.
    • Only change is the use of getPerfMetrics() accessor instead of a direct field for performance metrics.
  • Conclusion: No meaningful logic change.

b. submitPurgeKeysRequest

  • Moved to: KeyDeletingService
  • Diff/Change:
    Logic remains identical:
    • Handles building the PurgeKeysRequest, sets fields, error handling, and return values.
    • Uses getClientId().toString() instead of direct field, as with DirectoryDeletingService.
  • Conclusion: Only clientId accessor changed; logic is the same.

3. Functions Removed from AbstractKeyDeletingService (Not moved)

  • suspend, resume, shouldRun: Removed from both DirectoryDeletingService and KeyDeletingService as part of class cleanup.
    • No new version in the subclasses; these are simply no longer present.

Summary Table

Function Old Location New Location Changes?
optimizeDirDeletesAndSubmitRequest AbstractKeyDeletingService DirectoryDeletingService None
prepareDeleteDirRequest AbstractKeyDeletingService DirectoryDeletingService None
wrapPurgeRequest AbstractKeyDeletingService DirectoryDeletingService None
submitPurgePaths AbstractKeyDeletingService DirectoryDeletingService Only access modifier + clientId
getDeletedDirsCount AbstractKeyDeletingService DirectoryDeletingService None
getMovedDirsCount AbstractKeyDeletingService DirectoryDeletingService None
getMovedFilesCount AbstractKeyDeletingService DirectoryDeletingService None
processKeyDeletes AbstractKeyDeletingService KeyDeletingService None (minor accessor change)
submitPurgeKeysRequest AbstractKeyDeletingService KeyDeletingService Only clientId accessor

Conclusion

  • All functions were moved without logic changes (except minor access modifier and clientId accessor adjustments).
  • This refactor was a pure move to make base and subclass responsibilities clearer, with no impact on behavior or output.

If you want a detailed, line-by-line diff for a specific function, let me know the function name!

@swamirishi swamirishi marked this pull request as ready for review June 9, 2025 13:15
OzoneConfiguration configuration, int dirDeletingServiceCorePoolSize, boolean deepCleanSnapshots) {
super(DirectoryDeletingService.class.getSimpleName(), interval, unit,
dirDeletingServiceCorePoolSize, serviceTimeout, ozoneManager, null);
dirDeletingServiceCorePoolSize, serviceTimeout, ozoneManager);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

would dirDeletingServiceCorePoolSize needs to be larger than 10 if number of snapshots is big? should we pick a larger default value

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we can start out with default of 10. Snapshots would be always created lazily. Even if we create one snapshot in a minute. There would be at the most 20-30 snapshots for deep cleaning every 15 minutes rest of the snapshots would be just noop.

Copy link
Contributor

@jojochuang jojochuang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

A few suggestions made from my IDE. They can be dealt with in the future. For now I'd suggest to go ahead and merge it to make the refactor easier.


private static final class DeletedDirSupplier implements Closeable {
private TableIterator<String, ? extends KeyValue<String, OmKeyInfo>>
deleteTableIterator;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can be a final variable

return purgePathsRequest.build();
}

private OzoneManagerProtocolProtos.OMResponse submitPurgePaths(List<PurgePathRequest> requests,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

return value not used.

// Using multi thread for DirDeletion. Multiple threads would read
// from parent directory info from deleted directory table concurrently
// and send deletion requests.
private int ratisByteLimit;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can be made final

@swamirishi swamirishi merged commit f62eb8a into apache:master Jun 10, 2025
53 checks passed
@swamirishi
Copy link
Contributor Author

Thank you for reviewing the PR @jojochuang . I will create a follow up jira for the above review comments

jojochuang pushed a commit to jojochuang/ozone that referenced this pull request Jun 13, 2025
sadanand48 pushed a commit to sadanand48/hadoop-ozone that referenced this pull request Jun 16, 2025
aswinshakil added a commit to aswinshakil/ozone that referenced this pull request Jun 20, 2025
…239-container-reconciliation

Commits: 62

da53b5b HDDS-13299. Fix failures related to delete (apache#8665)
8c1b439 HDDS-13296. Integration check always passes due to missing output (apache#8662)
7329859 HDDS-13023. Container checksum is missing after container import (apache#8459)
a0af93e HDDS-13292. Change `<? extends KeyValue>` to `<KeyValue>` in test (apache#8657)
f3050cf HDDS-13276. Use KEY_ONLY/VALUE_ONLY iterator in SCM/Datanode. (apache#8638)
e9c0a45 HDDS-13262. Simplify key name validation (apache#8619)
f713e57 HDDS-12482. Avoid using CommonConfigurationKeys (apache#8647)
b574709 HDDS-12924. datanode used space calculation optimization (apache#8365)
de683aa HDDS-13263. Refactor DB Checkpoint Utilities. (apache#8620)
97262aa HDDS-13256. Updated OM Snapshot Grafana Dashboard to reflect metric updates from HDDS-13181. (apache#8639)
9d2b415 HDDS-13234. Expired secret key can abort leader OM startup. (apache#8601)
d9049a2 HDDS-13220. Change Recon 'Negative usedBytes' message loglevel to DEBUG (apache#8648)
6df3077 HDDS-9223. Use protobuf for SnapshotDiffJobCodec (apache#8503)
a7fc290 HDDS-13236. Change Table methods not to throw IOException. (apache#8645)
9958f5b HDDS-13287. Upgrade commons-beanutils to 1.11.0 due to CVE-2025-48734 (apache#8646)
48aefea HDDS-13277. [Docs] Native C/C++ Ozone clients (apache#8630)
052d912 HDDS-13037. Let container create command support STANDALONE , RATIS and EC containers (apache#8559)
90ed60b HDDS-13279. Skip verifying Apache Ranger binaries in CI (apache#8633)
9bc53b2 HDDS-11513. All deletion configurations should be configurable without restart (apache#8003)
ac511ac HDDS-13259. Deletion Progress - Grafana Dashboard (apache#8617)
3370f42 HDDS-13246. Change `<? extend KeyValue>` to `<KeyValue>` in hadoop-hdds (apache#8631)
7af8c44 HDDS-11454. Ranger integration for Docker Compose environment (apache#8575)
5a3e4e7 HDDS-13273. Bump awssdk to 2.31.63 (apache#8626)
77138b8 HDDS-13254. Change table iterator to optionally read key or value. (apache#8621)
ce288b6 HDDS-13265. Simplify the page Access Ozone using HTTPFS REST API (apache#8629)
36fe888 HDDS-13275. Improve CheckNative implementation (apache#8628)
d38484e HDDS-13274. Bump sqlite-jdbc to 3.50.1.0 (apache#8627)
3f3ec43 HDDS-13266. `ozone debug checknative` to show OpenSSL lib (apache#8623)
8983a63 HDDS-13272. Bump junit to 5.13.1 (apache#8625)
a927113 HDDS-13271. [Docs] Minor text updates, reference links. (apache#8624)
7e77058 HDDS-13112. [Docs] OM Bootstrap can also happen when follower falls behind too much. (apache#8600)
fd13300 HDDS-10775. Support bucket ownership verification (apache#8558)
3ecf345 HDDS-13207. [Docs] Third party systems compatible with Ozone S3. (apache#8584)
ad5a507 HDDS-13035. SnapshotDeletingService should hold write locks while purging deleted snapshots (apache#8554)
38a9186 HDDS-12637. Increase max buffer size for tar entry read/write (apache#8618)
f31c264 HDDS-13045. Implement Immediate Triggering of Heartbeat when Volume Full (apache#8590)
0701d6a HDDS-13248. Remove `ozone debug replicas verify` option --output-dir (apache#8612)
ca1afe8 HDDS-13257. Remove separate split for shell integration tests (apache#8616)
5d6fe94 HDDS-13216. Standardize Container[Replica]NotFoundException messages (apache#8599)
1e47217 HDDS-13168. Fix error response format in CheckUploadContentTypeFilter (apache#8614)
6d4d423 HDDS-13181. Added metrics for internal Snapshot Operations. (apache#8606)
4a461b2 HDDS-10490. Intermittent NPE in TestSnapshotDiffManager#testLoadJobsOnStartUp (apache#8596)
bf29f7f HDDS-13235. The equals/hashCode methods in anonymous KeyValue classes may not work. (apache#8607)
6ff3ad6 HDDS-12873. Improve ContainerData statistics synchronization. (apache#8305)
09d3b27 HDDS-13244. TestSnapshotDeletingServiceIntegrationTest should close snapshots after deleting them (apache#8611)
931bc2d HDDS-13243. copy-rename-maven-plugin version is missing (apache#8605)
3b5985c HDDS-13244. Disable TestSnapshotDeletingServiceIntegrationTest
6bf009c HDDS-12927. metrics and log to indicate datanode crossing disk limits (apache#8573)
752da2b HDDS-12760. Intermittent Timeout in testImportedContainerIsClosed (apache#8349)
8c32363 HDDS-13050. Update StartFromDockerHub.md. (apache#8586)
ba1887c HDDS-13241. Fix some potential resource leaks (apache#8602)
bbaf71e HDDS-13130. Rename all instances of Disk Usage to Namespace usage (apache#8571)
0628386 HDDS-13142. Correct SCMPerformanceMetrics for delete operation. (apache#8592)
516bc96 HDDS-13148. [Docs] Update Transparent Data Encryption doc. (apache#8530)
5787135 HDDS-13229. [Doc] Fix incorrect CLI argument order in OM upgrade docs (apache#8598)
ba95074 HDDS-13107. Support limiting output of `ozone admin datanode list` (apache#8595)
e7f5544 HDDS-13171. Replace pipelineID if nodes are changed (apache#8562)
3c9d4d8 HDDS-13103. Correct transaction metrics in SCMBlockDeletingService. (apache#8516)
f62eb8a HDDS-13160. Remove SnapshotDirectoryCleaningService and refactor AbstractDeletingService (apache#8547)
b46e6b2 HDDS-13150. Fixed SnapshotLimitCheck when failures occur. (apache#8532)
203c1d3 HDDS-13206. Update documentation for Apache Ranger (apache#8583)
2072ef0 HDDS-13214. populate-cache fails due to unused dependency (apache#8594)

Conflicts:
	hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/impl/ContainerData.java
	hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueContainer.java
	hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/helpers/KeyValueContainerUtil.java
	hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/statemachine/background/BlockDeletingTask.java
swamirishi added a commit to swamirishi/ozone that referenced this pull request Dec 3, 2025
…efactor AbstractDeletingService (apache#8547)

(cherry picked from commit f62eb8a)

 Conflicts:
	hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestOzoneManagerHASnapshot.java
	hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/service/TestRootedDDSWithFSO.java
	hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/service/AbstractKeyDeletingService.java
	hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/service/DirectoryDeletingService.java
	hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/service/TestKeyDeletingService.java

Change-Id: Icca49fb92b5cd20bd7159fce98262b8db791d4ec
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

snapshot https://issues.apache.org/jira/browse/HDDS-6517

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants