Skip to content

HDDS-12924. datanode used space calculation optimization #8365

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
ashishkumar50 merged 22 commits into from
Jun 18, 2025
Merged

HDDS-12924. datanode used space calculation optimization #8365

ashishkumar50 merged 22 commits into from
Jun 18, 2025

Conversation

@sumitagrawl
Copy link
Contributor

@sumitagrawl sumitagrawl commented Apr 30, 2025 •
edited
Loading

What changes were proposed in this pull request?

Approach 2 is concluded where used disk space is: sum(du over ozone path excluding container data, sum(all containers in memory size)

DUOptimized combine used values from:

  1. container data size from containerSet in Memory
  2. DU of volume path except Container datapath

As implementation,

  • OzoneContainer have method for getting sum of data size for all containers from ContainerSet for the volume.
  • This method as is passed as function to HddsVolume and then further callable to OptimizedDU to retrieve the overall data size for the volume

This way, dependency is removed as existing from OzoneContainer to StorageVolume, and then to SpaceUsages classes.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-12924

How was this patch tested?

  • Unit test cases added
  • Integration test cases added

@kerneltime
Copy link
Contributor

kerneltime commented May 6, 2025

running du at a higher frequency on a filesystem populated with thousands of containers and millions of blocks will be expensive. I prefer the approach in this doc where we always keep some min space free by including it in our calculation for disk used and base the disk used on container metatadata. We could optionally run df at a lower frequency to ensure the divergence between actual vs real is minimal (taking into account space reserved for other consumers of the volume)

@sumitagrawl
Copy link
Contributor Author

sumitagrawl commented May 7, 2025

As discussion over community meeting, Approach 2 is preferred, where ozone usages will be DU for metadata + summation of all container data size.

@sumitagrawl sumitagrawl requested review from ChenSammi and smengcl May 7, 2025 12:51
@sumitagrawl sumitagrawl marked this pull request as ready for review May 30, 2025 08:55
siddhantsangwan
siddhantsangwan approved these changes Jun 17, 2025
View reviewed changes
Copy link
Contributor

@siddhantsangwan siddhantsangwan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This code area has become complicated over time, but overall the changes LGTM.

ashishkumar50
ashishkumar50 reviewed Jun 18, 2025
View reviewed changes
Copy link
Contributor

@ashishkumar50 ashishkumar50 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sumitagrawl Thanks for the patch, please find comments inline.

ashishkumar50
ashishkumar50 approved these changes Jun 18, 2025
View reviewed changes
Copy link
Contributor

@ashishkumar50 ashishkumar50 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sumitagrawl Thanks for updating patch and explanation, LGTM.

@ashishkumar50 ashishkumar50 merged commit b574709 into apache:master Jun 18, 2025
42 checks passed
aswinshakil added a commit to aswinshakil/ozone that referenced this pull request Jun 20, 2025
@aswinshakil
Merge branch 'master' of https://github.com/apache/ozone into HDDS-10…
3c4b3be 
…239-container-reconciliation

Commits: 62

da53b5b HDDS-13299. Fix failures related to delete (apache#8665)
8c1b439 HDDS-13296. Integration check always passes due to missing output (apache#8662)
7329859 HDDS-13023. Container checksum is missing after container import (apache#8459)
a0af93e HDDS-13292. Change `<? extends KeyValue>` to `<KeyValue>` in test (apache#8657)
f3050cf HDDS-13276. Use KEY_ONLY/VALUE_ONLY iterator in SCM/Datanode. (apache#8638)
e9c0a45 HDDS-13262. Simplify key name validation (apache#8619)
f713e57 HDDS-12482. Avoid using CommonConfigurationKeys (apache#8647)
b574709 HDDS-12924. datanode used space calculation optimization (apache#8365)
de683aa HDDS-13263. Refactor DB Checkpoint Utilities. (apache#8620)
97262aa HDDS-13256. Updated OM Snapshot Grafana Dashboard to reflect metric updates from HDDS-13181. (apache#8639)
9d2b415 HDDS-13234. Expired secret key can abort leader OM startup. (apache#8601)
d9049a2 HDDS-13220. Change Recon 'Negative usedBytes' message loglevel to DEBUG (apache#8648)
6df3077 HDDS-9223. Use protobuf for SnapshotDiffJobCodec (apache#8503)
a7fc290 HDDS-13236. Change Table methods not to throw IOException. (apache#8645)
9958f5b HDDS-13287. Upgrade commons-beanutils to 1.11.0 due to CVE-2025-48734 (apache#8646)
48aefea HDDS-13277. [Docs] Native C/C++ Ozone clients (apache#8630)
052d912 HDDS-13037. Let container create command support STANDALONE , RATIS and EC containers (apache#8559)
90ed60b HDDS-13279. Skip verifying Apache Ranger binaries in CI (apache#8633)
9bc53b2 HDDS-11513. All deletion configurations should be configurable without restart (apache#8003)
ac511ac HDDS-13259. Deletion Progress - Grafana Dashboard (apache#8617)
3370f42 HDDS-13246. Change `<? extend KeyValue>` to `<KeyValue>` in hadoop-hdds (apache#8631)
7af8c44 HDDS-11454. Ranger integration for Docker Compose environment (apache#8575)
5a3e4e7 HDDS-13273. Bump awssdk to 2.31.63 (apache#8626)
77138b8 HDDS-13254. Change table iterator to optionally read key or value. (apache#8621)
ce288b6 HDDS-13265. Simplify the page Access Ozone using HTTPFS REST API (apache#8629)
36fe888 HDDS-13275. Improve CheckNative implementation (apache#8628)
d38484e HDDS-13274. Bump sqlite-jdbc to 3.50.1.0 (apache#8627)
3f3ec43 HDDS-13266. `ozone debug checknative` to show OpenSSL lib (apache#8623)
8983a63 HDDS-13272. Bump junit to 5.13.1 (apache#8625)
a927113 HDDS-13271. [Docs] Minor text updates, reference links. (apache#8624)
7e77058 HDDS-13112. [Docs] OM Bootstrap can also happen when follower falls behind too much. (apache#8600)
fd13300 HDDS-10775. Support bucket ownership verification (apache#8558)
3ecf345 HDDS-13207. [Docs] Third party systems compatible with Ozone S3. (apache#8584)
ad5a507 HDDS-13035. SnapshotDeletingService should hold write locks while purging deleted snapshots (apache#8554)
38a9186 HDDS-12637. Increase max buffer size for tar entry read/write (apache#8618)
f31c264 HDDS-13045. Implement Immediate Triggering of Heartbeat when Volume Full (apache#8590)
0701d6a HDDS-13248. Remove `ozone debug replicas verify` option --output-dir (apache#8612)
ca1afe8 HDDS-13257. Remove separate split for shell integration tests (apache#8616)
5d6fe94 HDDS-13216. Standardize Container[Replica]NotFoundException messages (apache#8599)
1e47217 HDDS-13168. Fix error response format in CheckUploadContentTypeFilter (apache#8614)
6d4d423 HDDS-13181. Added metrics for internal Snapshot Operations. (apache#8606)
4a461b2 HDDS-10490. Intermittent NPE in TestSnapshotDiffManager#testLoadJobsOnStartUp (apache#8596)
bf29f7f HDDS-13235. The equals/hashCode methods in anonymous KeyValue classes may not work. (apache#8607)
6ff3ad6 HDDS-12873. Improve ContainerData statistics synchronization. (apache#8305)
09d3b27 HDDS-13244. TestSnapshotDeletingServiceIntegrationTest should close snapshots after deleting them (apache#8611)
931bc2d HDDS-13243. copy-rename-maven-plugin version is missing (apache#8605)
3b5985c HDDS-13244. Disable TestSnapshotDeletingServiceIntegrationTest
6bf009c HDDS-12927. metrics and log to indicate datanode crossing disk limits (apache#8573)
752da2b HDDS-12760. Intermittent Timeout in testImportedContainerIsClosed (apache#8349)
8c32363 HDDS-13050. Update StartFromDockerHub.md. (apache#8586)
ba1887c HDDS-13241. Fix some potential resource leaks (apache#8602)
bbaf71e HDDS-13130. Rename all instances of Disk Usage to Namespace usage (apache#8571)
0628386 HDDS-13142. Correct SCMPerformanceMetrics for delete operation. (apache#8592)
516bc96 HDDS-13148. [Docs] Update Transparent Data Encryption doc. (apache#8530)
5787135 HDDS-13229. [Doc] Fix incorrect CLI argument order in OM upgrade docs (apache#8598)
ba95074 HDDS-13107. Support limiting output of `ozone admin datanode list` (apache#8595)
e7f5544 HDDS-13171. Replace pipelineID if nodes are changed (apache#8562)
3c9d4d8 HDDS-13103. Correct transaction metrics in SCMBlockDeletingService. (apache#8516)
f62eb8a HDDS-13160. Remove SnapshotDirectoryCleaningService and refactor AbstractDeletingService (apache#8547)
b46e6b2 HDDS-13150. Fixed SnapshotLimitCheck when failures occur. (apache#8532)
203c1d3 HDDS-13206. Update documentation for Apache Ranger (apache#8583)
2072ef0 HDDS-13214. populate-cache fails due to unused dependency (apache#8594)

Conflicts:
	hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/impl/ContainerData.java
	hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueContainer.java
	hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/helpers/KeyValueContainerUtil.java
	hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/statemachine/background/BlockDeletingTask.java
swamirishi pushed a commit to swamirishi/ozone that referenced this pull request Dec 3, 2025
@sumitagrawl
CDPD-82370. HDDS-12924. datanode used space calculation optimization (a…
0f60856 
…pache#8365)

(cherry picked from commit b574709)

Conflicts:
StorageVolume and VolumeInfo: The StorageVolume(Builder<?> b) method had conflicts since HDDS-12679 (Merge VolumeInfo into StorageVolume) is not present in 7.1.9. Resolved conflict by adding exclusionProvider and containerUsedSpace to the Builder of VolumeInfo. Edited VolumeInfo(Builder b) to pass exclusionProvider to usageCheckFactory.paramsFor(root, b.exclusionProvider) and then setting containerUsedSpace by checkParams.setContainerUsedSpace(b.containerUsedSpace).

TestDatanodeStateMachine: Fixed an unrelated test that was brought in by HDDS-12547 (Container creation and import use the same VolumeChoosingPolicy). We continue to use RoundRobinVolumeChoosingPolicy but the test was expecting CapacityVolumeChoosingPolicy.

	hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/fs/DU.java
	hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/fs/DUOptimized.java
	hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/fs/DUOptimizedFactory.java
	hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/fs/SpaceUsageCheckFactory.java
	hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/fs/SpaceUsageCheckParams.java
	hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/fs/TestDUOptimized.java
	hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/fs/TestDUOptimizedFactory.java
	hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/HddsVolume.java
	hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/MutableVolumeSet.java
	hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/StorageVolume.java
	hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/ozoneimpl/OzoneContainer.java
	hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/impl/TestHddsDispatcher.java
	hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/volume/TestHddsVolume.java
	hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/ozoneimpl/TestOzoneContainer.java
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@siddhantsangwan siddhantsangwan siddhantsangwan approved these changes

@ashishkumar50 ashishkumar50 ashishkumar50 approved these changes

@kerneltime kerneltime Awaiting requested review from kerneltime

@umamaheswararao umamaheswararao Awaiting requested review from umamaheswararao

@nandakumar131 nandakumar131 Awaiting requested review from nandakumar131

@errose28 errose28 Awaiting requested review from errose28

@smengcl smengcl Awaiting requested review from smengcl

@ChenSammi ChenSammi Awaiting requested review from ChenSammi

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@sumitagrawl @kerneltime @ChenSammi @siddhantsangwan @smengcl @ashishkumar50