HDDS-4925. Introduce ContainerBalancer in SCM with start/stop capabilities. #2097
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
lokeshj1703
reviewed
Apr 1, 2021
Contributor
lokeshj1703
left a comment
lokeshj1703
left a comment
There was a problem hiding this comment.
@siddhantsangwan Thanks for working on this! The changes look good to me. I have few minor comments inline.
| */ | ||
| @ConfigGroup(prefix = "hdds.container.balancer") | ||
| public final class ContainerBalancerConfiguration { | ||
| @Config(key = "threshold", type = ConfigType.AUTO, defaultValue = "0.1", |
Contributor
There was a problem hiding this comment.
Let's have a prefix "utilisation" or sth more suitable for this key.
| "threshold value.") | ||
| private double threshold; | ||
|
|
||
| @Config(key = "max.datanodes.balanced", type = ConfigType.INT, |
Contributor
There was a problem hiding this comment.
Lets rename the key to datanodes.balanced.max. max can be suffix. Similarly for the other config.
# Conflicts: # hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/StorageContainerManager.java
Contributor
Author
|
I have incorporated the suggestions and made some other changes to pass the findbugs check. Please let me know if there are any more suggestions. |
lokeshj1703
approved these changes
Apr 7, 2021
Contributor
lokeshj1703
left a comment
lokeshj1703
left a comment
There was a problem hiding this comment.
@siddhantsangwan Thanks for updating the PR! The changes look good to me. +1.
lokeshj1703
pushed a commit
that referenced
this pull request
Apr 7, 2021
Contributor
|
@siddhantsangwan Thanks for the contribution! I have committed the PR to master branch. |
errose28
added a commit
to errose28/ozone
that referenced
this pull request
Apr 7, 2021
* HDDS-3698-nonrolling-upgrade: HDDS-5056. Avoid false positiver error messages during pipeline creations (apache#2105) HDDS-5027. [SCM HA Security] Handle leader changes during bootstrap. (apache#2113) HDDS-5032. Fix findbugs (apache#2120) HDDS-5062. Add a config to bypass clusterId validation for bootstrapping SCM. (apache#2114) HDDS-5011. Introduce Java based ReplicationConfig implementation (apache#2089) HDDS-4925. Introduce ContainerBalancer in SCM with start/stop capabilities. (apache#2097)
errose28
added a commit
to errose28/ozone
that referenced
this pull request
Apr 7, 2021
* HDDS-3698-nonrolling-upgrade: HDDS-5056. Avoid false positiver error messages during pipeline creations (apache#2105) HDDS-5027. [SCM HA Security] Handle leader changes during bootstrap. (apache#2113) HDDS-5032. Fix findbugs (apache#2120) HDDS-5062. Add a config to bypass clusterId validation for bootstrapping SCM. (apache#2114) HDDS-5011. Introduce Java based ReplicationConfig implementation (apache#2089) HDDS-4925. Introduce ContainerBalancer in SCM with start/stop capabilities. (apache#2097)
errose28
added a commit
to errose28/ozone
that referenced
this pull request
Apr 9, 2021
* HDDS-3698-nonrolling-upgrade: (150 commits) HDDS-5056. Avoid false positiver error messages during pipeline creations (apache#2105) HDDS-5027. [SCM HA Security] Handle leader changes during bootstrap. (apache#2113) HDDS-5032. Fix findbugs (apache#2120) HDDS-5062. Add a config to bypass clusterId validation for bootstrapping SCM. (apache#2114) HDDS-5011. Introduce Java based ReplicationConfig implementation (apache#2089) HDDS-4925. Introduce ContainerBalancer in SCM with start/stop capabilities. (apache#2097) fix project name in NOTICE.txt (apache#2112) HDDS-5066. Use fixed vesion from pnpm to build recon (apache#2115) HDDS-5014. Add non-rolling upgrade design docs. HDDS-5035. Use default config values to solve generated config file conflict (apache#2087) HDDS-5032. DN stopped to load containers on volume after a container load exception. (apache#2109) HDDS-4504. Datanode deletion config should be based on number of blocks (apache#1885) Fix ozone-ha acceptance test. HDDS-5058. Make getScmInfo retry for a duration. HDDS-4506. Support query parameter based v4 auth in S3g (apache#1628) HDDS-4553. ChunkInputStream should release buffer as soon as last byte in the buffer is read (apache#2062) HDDS-5022. SCM get roles command should provide Ratis Leader/Follower… (apache#2098) HDDS-5033. SCM may not be able to know full port list of Datanode after Datanode is started. (apache#2090) HDDS-3752. Fix o3fs list bucket contents issue when without tailing "/" (apache#2088) HDDS-4901. Remove OmOzoneAclMap from OmVolumeArgs to avoid OzoneAcl conversions (apache#1992) ...
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Introducing
ContainerBalancerwith start and stop methods.ContainerBalancerConfigurationclass contains the necessary config values. AddingBALANCERtag inConfigTag.ContainerBalanceris constructed whenStorageContainerManageris created.What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-4925
How was this patch tested?
TestStorageContainerManagerHA