Skip to content

YARN-11646. Do not ignore zero memory capacity config in QueueCapacityConfigParser. #6433

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
brumi1024 merged 2 commits into from
Jan 11, 2024
Merged

YARN-11646. Do not ignore zero memory capacity config in QueueCapacityConfigParser. #6433

brumi1024 merged 2 commits into from
Jan 11, 2024

Conversation

@tomicooler
Copy link
Contributor

@tomicooler tomicooler commented Jan 10, 2024

Change-Id: Id00a5382d982dcf33ec8a3769fc33419b660968b

Description of PR

There is no reason to ignore the configured capacity if the memory is 0 in the configuration.

It makes it impossible to configure a zero absolute resource capacity.

Example:

root.default.capacity=[memory=0, vcores=0]
root.default.maximum-capacity=[memory=2048, vcores=2]

How was this patch tested?

Ran the unit tests.

For code changes:

  • Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'YARN-11646. Your PR title ...')?
  • Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
  • If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
  • If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

@tomicooler
YARN-11646. Do not ignore zero memory capacity config in QueueCapacit…
29d5dd4 
…yConfigParser.

Change-Id: Id00a5382d982dcf33ec8a3769fc33419b660968b
@hadoop-yetus
Copy link

hadoop-yetus commented Jan 10, 2024

🎊 +1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 21s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ trunk Compile Tests _
+1 💚 mvninstall 32m 45s trunk passed
+1 💚 compile 0m 31s trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚 compile 0m 27s trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚 checkstyle 0m 29s trunk passed
+1 💚 mvnsite 0m 30s trunk passed
+1 💚 javadoc 0m 33s trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚 javadoc 0m 27s trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚 spotbugs 1m 4s trunk passed
+1 💚 shadedclient 20m 44s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 0m 26s the patch passed
+1 💚 compile 0m 27s the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚 javac 0m 27s the patch passed
+1 💚 compile 0m 24s the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚 javac 0m 24s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
-0 ⚠️ checkstyle 0m 20s /results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
+1 💚 mvnsite 0m 26s the patch passed
+1 💚 javadoc 0m 24s the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚 javadoc 0m 24s the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚 spotbugs 1m 9s the patch passed
+1 💚 shadedclient 19m 36s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 84m 56s hadoop-yarn-server-resourcemanager in the patch passed.
+1 💚 asflicense 0m 21s The patch does not generate ASF License warnings.
168m 38s
Subsystem Report/Notes
Docker ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6433/1/artifact/out/Dockerfile
GITHUB PR #6433
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux 0d3fc268fb08 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 29d5dd4
Default Java Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6433/1/testReport/
Max. process+thread count 953 (vs. ulimit of 5500)
modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6433/1/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

hadoop-yetus commented Jan 10, 2024

🎊 +1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 21s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ trunk Compile Tests _
+1 💚 mvninstall 31m 43s trunk passed
+1 💚 compile 0m 33s trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚 compile 0m 28s trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚 checkstyle 0m 29s trunk passed
+1 💚 mvnsite 0m 31s trunk passed
+1 💚 javadoc 0m 33s trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚 javadoc 0m 28s trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚 spotbugs 1m 8s trunk passed
+1 💚 shadedclient 19m 40s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 0m 26s the patch passed
+1 💚 compile 0m 26s the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚 javac 0m 26s the patch passed
+1 💚 compile 0m 23s the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚 javac 0m 23s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
-0 ⚠️ checkstyle 0m 20s /results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
+1 💚 mvnsite 0m 25s the patch passed
+1 💚 javadoc 0m 24s the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚 javadoc 0m 24s the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚 spotbugs 1m 5s the patch passed
+1 💚 shadedclient 19m 37s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 86m 5s hadoop-yarn-server-resourcemanager in the patch passed.
+1 💚 asflicense 0m 23s The patch does not generate ASF License warnings.
167m 31s
Subsystem Report/Notes
Docker ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6433/2/artifact/out/Dockerfile
GITHUB PR #6433
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux da84b7c3b3c3 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 29d5dd4
Default Java Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6433/2/testReport/
Max. process+thread count 941 (vs. ulimit of 5500)
modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6433/2/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@tomicooler
fix checkstyle
60444ba 
Change-Id: I4dd06e96cc6bb2ac63f9ef3062bba3e8d0f8509e
@hadoop-yetus
Copy link

hadoop-yetus commented Jan 11, 2024

🎊 +1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 20s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 1s codespell was not available.
+0 🆗 detsecrets 0m 1s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ trunk Compile Tests _
+1 💚 mvninstall 31m 30s trunk passed
+1 💚 compile 0m 33s trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚 compile 0m 30s trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚 checkstyle 0m 30s trunk passed
+1 💚 mvnsite 0m 32s trunk passed
+1 💚 javadoc 0m 35s trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚 javadoc 0m 26s trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚 spotbugs 1m 10s trunk passed
+1 💚 shadedclient 19m 52s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 0m 25s the patch passed
+1 💚 compile 0m 26s the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚 javac 0m 26s the patch passed
+1 💚 compile 0m 24s the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚 javac 0m 24s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 checkstyle 0m 21s the patch passed
+1 💚 mvnsite 0m 27s the patch passed
+1 💚 javadoc 0m 24s the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚 javadoc 0m 24s the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚 spotbugs 1m 8s the patch passed
+1 💚 shadedclient 19m 49s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 86m 3s hadoop-yarn-server-resourcemanager in the patch passed.
+1 💚 asflicense 0m 24s The patch does not generate ASF License warnings.
167m 50s
Subsystem Report/Notes
Docker ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6433/3/artifact/out/Dockerfile
GITHUB PR #6433
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux cc5e70693e1e 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 60444ba
Default Java Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6433/3/testReport/
Max. process+thread count 952 (vs. ulimit of 5500)
modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6433/3/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@K0K0V0K
Copy link
Contributor

K0K0V0K commented Jan 11, 2024

Thanks @tomicooler for this change! LGTM!

brumi1024
brumi1024 approved these changes Jan 11, 2024
View reviewed changes
Copy link
Member

@brumi1024 brumi1024 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @tomicooler for the patch, @K0K0V0K for the review. LGTM, merging to trunk.

@brumi1024 brumi1024 merged commit 55b9f87 into apache:trunk Jan 11, 2024
@brumi1024 brumi1024 mentioned this pull request Jan 25, 2024
Merged
4 tasks
jiajunmao pushed a commit to jiajunmao/hadoop-MLEC that referenced this pull request Feb 6, 2024
@tomicooler @jiajunmao
YARN-11646. Do not ignore zero memory capacity config in QueueCapacit…
41df6e0 
…yConfigParser. (apache#6433)
susheelgupta7 pushed a commit to susheelgupta7/hadoop that referenced this pull request Aug 1, 2025
@tomicooler @susheelgupta7
YARN-11646. Do not ignore zero memory capacity config in QueueCapacit…
e5471d7 
…yConfigParser. (apache#6433)
susheelgupta7 pushed a commit to susheelgupta7/hadoop that referenced this pull request Aug 5, 2025
@tomicooler @susheelgupta7
YARN-11646. Do not ignore zero memory capacity config in QueueCapacit…
0284fb9 
…yConfigParser. (apache#6433)
steveloughran pushed a commit to steveloughran/hadoop that referenced this pull request Aug 5, 2025
@tomicooler @steveloughran
COMPX-15960: YARN-11646. Do not ignore zero memory capacity config in…
192c7c9 
… QueueCapacityConfigParser. (apache#6433)

Change-Id: Ibbecee0193bb9af09df4d227bc52efbc322a37c7
(cherry picked from commit 436ef1676ec0f083fc0d7057f902569cecb25ebb)
susheelgupta7 pushed a commit to susheelgupta7/hadoop that referenced this pull request Aug 7, 2025
@tomicooler @susheelgupta7
YARN-11646. Do not ignore zero memory capacity config in QueueCapacit…
92e475f 
…yConfigParser. (apache#6433)
susheelgupta7 pushed a commit to susheelgupta7/hadoop that referenced this pull request Aug 30, 2025
@tomicooler @susheel-gupta
YARN-11646. Do not ignore zero memory capacity config in QueueCapacit…
475fd3f 
…yConfigParser. (apache#6433)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@brumi1024 brumi1024 brumi1024 approved these changes

Assignees

No one assigned

Labels

trunk YARN

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@tomicooler @hadoop-yetus @K0K0V0K @brumi1024