Fix weighted fair scheduling in disagg coordinator #18543
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
...o-main/src/main/java/com/facebook/presto/execution/resourceGroups/InternalResourceGroup.java
Outdated
Show resolved
Hide resolved
swapsmagic
force-pushed
the
disagg_weighted_fair_fix
branch
2 times, most recently
from
c5c51c1 to
4fb3642
Compare
swapsmagic
commented
Oct 21, 2022
...o-main/src/main/java/com/facebook/presto/execution/resourceGroups/InternalResourceGroup.java
Outdated
Show resolved
Hide resolved
swapsmagic
force-pushed
the
disagg_weighted_fair_fix
branch
from
4fb3642 to
59c4a8b
Compare
In disagg coordinator, each coorinator is considering it's own running queries to decide which resource group should run the next query in weighted fair scheduling. This lead to an issue when with n coordinators each allowing single query to run for the resource group which has lower weight and lead to n slots been used by the resource group even if it's weight will allow it to run lower than n queries. With the fix, we consider global running query count for the given resource group, which helps decide the correct number of queries we should run for a lower weight resource group and allow more slots be given to high priority resource group as per weighted fair scheduling.
swapsmagic
force-pushed
the
disagg_weighted_fair_fix
branch
from
59c4a8b to
2aa0b44
Compare
tdcmeehan
approved these changes
Oct 24, 2022
30 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Test plan - unit test + shadow run