kvserver: track priority inversion in replicate queue metrics #152624
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Member
|
This change is |
wenyihu6
force-pushed
the
newobs
branch
5 times, most recently
from
14f55e5 to
d80f84e
Compare
wenyihu6
changed the title
wenyihu6
changed the title
sumeerbhola
reviewed
Aug 28, 2025
Collaborator
sumeerbhola
left a comment
sumeerbhola
left a comment
There was a problem hiding this comment.
@sumeerbhola reviewed 1 of 2 files at r3, all commit messages.
Reviewable status:complete! 0 of 0 LGTMs obtained (waiting on @arulajmani and @tbg)
pkg/kv/kvserver/replicate_queue.go line 322 at r3 (raw file):
Unit: metric.Unit_COUNT, } metaReplicateQueuePriorityInversionForAddVoterCount = metric.Metadata{
(drive-by comments)
- Do we need all the detailed metrics below. Having too many metrics can be a cost. If needed, we should consider using labeled metrics, so that they are easy to deal with in systems that allow labels https://docs.google.com/document/d/1fn5KX0uPUCw1uW4KkCvmP9-svga0F6XSvhnqdg24MpE/edit?tab=t.0#heading=h.9ndo02nsmu
- Do we already have an enqueue counter metric, that one could compare the rate with of the requeue metric? For instance a requeue rate of 1/s may not matter if the enqueue rate is 100/s.
This commit adds a new cluster setting PriorityInversionRequeue that controls whether the replicate queue should requeue replicas when their priority at enqueue time differs significantly from their priority at processing time (e.g. dropping from top 3 to the lowest priority).
Previously, a replica could enter the queue with high priority but, by the time it was processed, the action planned for this replica may have a low priority, causing us to perform low priority work. Specifically, we are mostly worried about cases where the priority changes from any of the repair actions to consider rebalance. Rebalancing could take a long time and block other ranges enqueued with actual repair action needed. This commit ensures that such replicas are requeued instead, avoiding priority inversions.
Previously, replicas could be enqueued at a high priority but end up processing a lower-priority actions, causing priority inversion and unfairness to other replicas behind them that needs a repair action. This commit adds metrics to track such cases. In addition, this commit also adds metrics to track when replicas are requeued in the replicate queue due to a priority inversion from a repair action to a rebalance action.
Contributor
Author
|
Closing in favour of a new PR. Will address your comments in the new PR. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Stacked on top of #152596.
Previously, replicas could be enqueued at a high priority but end up processing
a lower-priority actions, causing priority inversion and unfairness to other
replicas behind them that needs a repair action. This commit adds metrics to
track such cases. In addition, this commit also adds metrics to track when
replicas are requeued in the replicate queue due to a priority inversion from a
repair action to a rebalance action.
Part of: #151847
Release note: none