Skip to content

YARN-10949. Simplify AbstractCSQueue#updateMaxAppRelatedField and fin… #3500

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
shuzirra merged 7 commits into from
Oct 20, 2021
Merged

YARN-10949. Simplify AbstractCSQueue#updateMaxAppRelatedField and fin… #3500

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
f3bc59c
YARN-10949. Simplify AbstractCSQueue#updateMaxAppRelatedField and fin…
9uapaw Sep 29, 2021
e5d0619
YARN-10949. Fix checkstyle errors
9uapaw Sep 30, 2021
affcba2
YARN-10949. Maintain backward compatibility in max apps calculation
9uapaw Oct 4, 2021
cb26800
YARN-10949. Limit global and system max app
9uapaw Oct 18, 2021
4b46cc7
YARN-10949. Fix app limit and test
9uapaw Oct 18, 2021
5220863
YARN-10949. Add test case for max system app trimming
9uapaw Oct 18, 2021
3edeac3
YARN-10949. Fix ApplicationLimit test
9uapaw Oct 20, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
47 changes: 0 additions & 47 deletions ...ava/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractCSQueue.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1475,53 +1475,6 @@ private Resource getOrInheritMaxResource(Resource resourceByLabel, String label)
configuredMaxResource, parentMaxResource));
}

void updateMaxAppRelatedField(CapacitySchedulerConfiguration conf,
LeafQueue leafQueue) {
int maxApplications = conf.getMaximumApplicationsPerQueue(queuePath);
int maxGlobalPerQueueApps = conf.getGlobalMaximumApplicationsPerQueue();
String maxLabel = RMNodeLabelsManager.NO_LABEL;

if (maxApplications < 0) {
for (String label : configuredNodeLabels) {
int maxApplicationsByLabel = 0;
if (maxGlobalPerQueueApps > 0) {
// In absolute mode, should
// shrink when change to corresponding label capacity.
maxApplicationsByLabel = this.capacityConfigType
!= CapacityConfigType.ABSOLUTE_RESOURCE ?
maxGlobalPerQueueApps :
(int) (maxGlobalPerQueueApps * queueCapacities
.getAbsoluteCapacity(label));
} else {
maxApplicationsByLabel = (int) (conf.getMaximumSystemApplications()
* queueCapacities.getAbsoluteCapacity(label));
}
if (maxApplicationsByLabel > maxApplications) {
maxApplications = maxApplicationsByLabel;
maxLabel = label;
}
}
}
leafQueue.setMaxApplications(maxApplications);

int maxApplicationsPerUser = Math.min(maxApplications,
(int) (maxApplications
* (leafQueue.getUsersManager().getUserLimit() / 100.0f)
* leafQueue.getUsersManager().getUserLimitFactor()));
if (leafQueue.getUsersManager().getUserLimitFactor() == -1) {
maxApplicationsPerUser = maxApplications;
}

leafQueue.setMaxApplicationsPerUser(maxApplicationsPerUser);
LOG.info("LeafQueue:" + leafQueue.getQueuePath() +
"update max app related, maxApplications="
+ maxApplications + ", maxApplicationsPerUser="
+ maxApplicationsPerUser + ", Abs Cap:" + queueCapacities
.getAbsoluteCapacity(maxLabel) + ", Cap: " + queueCapacities
.getCapacity(maxLabel) + ", MaxCap : " + queueCapacities
.getMaximumCapacity(maxLabel));
}

void deriveCapacityFromAbsoluteConfigurations(String label,
Resource clusterResource) {
// Update capacity with a float calculated from the parent's minResources
Expand Down
57 changes: 54 additions & 3 deletions ...main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,6 @@

import java.io.IOException;
import java.util.*;
import java.util.Map.Entry;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ConcurrentMap;

Expand Down Expand Up @@ -1939,8 +1938,9 @@ public void updateClusterResource(Resource clusterResource,
updateAbsoluteCapacities();

super.updateEffectiveResources(clusterResource);
super.updateMaxAppRelatedField(csContext.getConfiguration(),
this);

// Update maximum applications for the queue and for users
updateMaximumApplications(csContext.getConfiguration());

updateCurrentResourceLimits(currentResourceLimits, clusterResource);

Expand Down Expand Up @@ -2326,6 +2326,57 @@ public void stopQueue() {
}
}

void updateMaximumApplications(CapacitySchedulerConfiguration conf) {
int maxAppsForQueue = conf.getMaximumApplicationsPerQueue(getQueuePath());

int maxGlobalApplications = conf.getGlobalMaximumApplicationsPerQueue();
int maxSystemApplications = conf.getMaximumSystemApplications();
int baseMaxApplications = maxGlobalApplications > 0 ?
Comment thread
9uapaw marked this conversation as resolved.
Outdated
maxGlobalApplications : maxSystemApplications;

@shuzirra shuzirra Oct 7, 2021

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think, this should be more like a Math.min situation, if the maxGlobal is defined, it is a per queue limit, while max System is the total number of applications, so if System is set, it should be always a limiting factor. Also if System is too high, then maxGlobal should be the limiting factor, so it's more like

maxGlobalApplications > 0 ? Math.min(maxGlobalApplications, maxSystemApplications) : maxSystemApplications;

@9uapaw 9uapaw Oct 18, 2021

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree on this and this change still conforms to our existing tests.

@9uapaw 9uapaw Oct 18, 2021

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The problem with this approach is that system max app defaults to 10000. This means you basically can not set maxGlobalApp higher than this value, because it will be trimmed to 10000. We could pursue this path by making maxSystemApp defaults to -1 and check if it is defined, but I am not convinced it is worth the effort.

@shuzirra shuzirra Oct 18, 2021

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

According to the documentation:
Maximum number of applications in the system which can be concurrently active both running and pending. Limits on each queue are directly proportional to their queue capacities and user limits. This is a hard limit and any applications submitted when this limit is reached will be rejected. Default is 10000
This means to me, that maxSystemApp should be an absolute upper limit, if the user wishes to go above it, they should raise this limit. MaxGlobalApp should be always <= maxSystem app. But look at this the following way: if the user has 100 queues with maxGlobalApp 1000, and they saturate 10 of them with 10000 applications, then they won't be able to start any application in the 11th queue, because of the maxSystem is reached. So this limitation is already in place. As it should be.
See:

hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java

Line 1044 in 5337beb

if (isSystemAppsLimitReached()) {

So capacity scheduler will enforce the max system application anyway, even if the leaf queue would allow it, so there is no point in allowing more application per leaf queue than the system wide max applications.


String maxLabel = RMNodeLabelsManager.NO_LABEL;
if (maxAppsForQueue < 0) {
if (maxGlobalApplications > 0 && this.capacityConfigType
!= CapacityConfigType.ABSOLUTE_RESOURCE) {
maxAppsForQueue = maxGlobalApplications;
} else {
for (String label : configuredNodeLabels) {
int maxApplicationsByLabel = (int) (baseMaxApplications
* queueCapacities.getAbsoluteCapacity(label));
if (maxApplicationsByLabel > maxAppsForQueue) {
maxAppsForQueue = maxApplicationsByLabel;
maxLabel = label;
}
}
}
}

setMaxApplications(maxAppsForQueue);

updateMaxAppsPerUser();

LOG.info("LeafQueue:" + getQueuePath() +
"update max app related, maxApplications="
+ maxAppsForQueue + ", maxApplicationsPerUser="
+ maxApplicationsPerUser + ", Abs Cap:" + queueCapacities
.getAbsoluteCapacity(maxLabel) + ", Cap: " + queueCapacities
.getCapacity(maxLabel) + ", MaxCap : " + queueCapacities
.getMaximumCapacity(maxLabel));
}

private void updateMaxAppsPerUser() {
int maxAppsPerUser = maxApplications;
if (getUsersManager().getUserLimitFactor() != -1) {
int maxApplicationsWithUserLimits = (int) (maxApplications
* (getUsersManager().getUserLimit() / 100.0f)
* getUsersManager().getUserLimitFactor());
maxAppsPerUser = Math.min(maxApplications,
maxApplicationsWithUserLimits);
}

setMaxApplicationsPerUser(maxAppsPerUser);
}

/**
* Get all valid users in this queue.
* @return user list
Expand Down