HDDS-6901. Configure HDDS volume reserved as percentage of the volume space. #3532
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
kerneltime
reviewed
Jun 24, 2022
...tainer-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/VolumeInfo.java
Outdated
Show resolved
Hide resolved
errose28
reviewed
Jun 25, 2022
Contributor
errose28
left a comment
errose28
left a comment
There was a problem hiding this comment.
Thanks for working on this @aswinshakil. I've added some comments in addition to Ritesh.
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/ScmConfigKeys.java
Outdated
Show resolved
Hide resolved
...tainer-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/VolumeInfo.java
Outdated
Show resolved
Hide resolved
...tainer-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/VolumeInfo.java
Show resolved
Hide resolved
...tainer-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/VolumeInfo.java
Outdated
Show resolved
Hide resolved
kerneltime
reviewed
Jun 27, 2022
| continue; | ||
| } | ||
| //Both the configs are set. Log it and return 0 | ||
| if (reserveList.size() > 0 && percentage != defaultValue) { |
Contributor
There was a problem hiding this comment.
Should the behavior be to pick the max of the 2 values? Would be less problematic than not honoring any reserved space?
Member
Author
There was a problem hiding this comment.
The behavior should be set to only one config. It is either by each volume or for all volumes.
kerneltime
reviewed
Jun 27, 2022
...tainer-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/VolumeInfo.java
Show resolved
Hide resolved
kerneltime
reviewed
Jun 30, 2022
hadoop-hdds/config/src/main/java/org/apache/hadoop/hdds/conf/ConfigurationSource.java
Show resolved
Hide resolved
kerneltime
reviewed
Jun 30, 2022
...tainer-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/VolumeInfo.java
Outdated
Show resolved
Hide resolved
Contributor
|
Thank you @aswinshakil for the contribution. |
errose28
added a commit
to errose28/ozone
that referenced
this pull request
Jul 12, 2022
* master: (46 commits) HDDS-6901. Configure HDDS volume reserved as percentage of the volume space. (apache#3532) HDDS-6978. EC: Cleanup RECOVERING container on DN restarts (apache#3585) HDDS-6982. EC: Attempt to cleanup the RECOVERING container when reconstruction failed at coordinator. (apache#3583) HDDS-6968. Addendum: [Multi-Tenant] Fix USER_MISMATCH error even on correct user. (apache#3578) HDDS-6794. EC: Analyze and add putBlock even on non writing node in the case of partial single stripe. (apache#3514) HDDS-6900. Propagate TimeoutException for all SCM HA Ratis calls. (apache#3564) HDDS-6938. handle NPE when removing prefixAcl (apache#3568) HDDS-6960. EC: Implement the Over-replication Handler (apache#3572) HDDS-6979. Remove unused plexus dependency declaration (apache#3579) HDDS-6957. EC: ReplicationManager - priortise under replicated containers (apache#3574) HDDS-6723. Close Rocks objects properly in OzoneManager (apache#3400) HDDS-6942. Ozone Buckets/Objects created via S3 should not allow group access (apache#3553) HDDS-6965. Increase timeout for basic check (apache#3563) HDDS-6969. Add link to compose directory in smoketest README (apache#3567) HDDS-6970. EC: Ensure DatanodeAdminMonitor can handle EC containers during decommission (apache#3573) HDDS-6977. EC: Remove references to ContainerReplicaPendingOps in TestECContainerReplicaCount (apache#3575) HDDS-6217. Cleanup XceiverClientGrpc TODOs, and document how the client works and should be used. (apache#3012) HDDS-6773. Cleanup TestRDBTableStore (apache#3434) - fix checkstyle HDDS-6773. Cleanup TestRDBTableStore (apache#3434) HDDS-6676. KeyValueContainerData#getProtoBufMessage() should set block count (apache#3371) ... Conflicts: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/upgrade/SCMUpgradeFinalizer.java
duongkame
pushed a commit
to duongkame/ozone
that referenced
this pull request
Aug 16, 2022
…f the volume space. apache#3532 Change-Id: Ifd904a9213c4ef18feed91879a15eea5ba6ea5ee
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Currently, We can reserve space on a volume using
hdds.datanode.dir.du.reserved, which is key-value pair where we specify the volume and the space that should be reserved in that volume (i.e data1:5000MB). If there are multiple volumes we have put a key-value pair of volume:reserved for each volume. This PR aims to create a configuration that sets the percentage of volume reserved for all the volumes in the Datanode.What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-6901
How was this patch tested?
Patch was tested manually using docker and unit tests.