Improve scale writer creation based on producer buffer#14193
Merged
wenleix merged 1 commit intoprestodb:masterfrom Mar 6, 2020
Merged
Improve scale writer creation based on producer buffer#14193wenleix merged 1 commit intoprestodb:masterfrom
wenleix merged 1 commit intoprestodb:masterfrom
Conversation
46691e3 to
fc453bb
Compare
arhimondr
reviewed
Mar 4, 2020
Member
There was a problem hiding this comment.
I'm a little bit confused about this loging. I was thinking more about changing the condition to something like if 50% overutilized OR 90% non empty - add more writer. Thoughts?
Contributor
Author
There was a problem hiding this comment.
@arhimondr : Addressed the comments per offline discussion. Basically now we add more writers if the overall buffer utilization is more than the number of writers.
We don't need to worry about accidentally open too many writers since overall writtenBytes has to be more than writerMinSizeBytes * scheduledNodes.size()
writtenBytes >= writerMinSizeBytes * scheduledNodes.size())
arhimondr
approved these changes
Mar 5, 2020
We observe two cases in production cause current scale writer heuristics not able to scale: 1. When there is skew on the producer side and more than half of the producer buffer is not overutilized. 2. When grouped execution is enabled and each bucket doesn't make the buffer to be overutilized. This commit tries to improve the situation by considering overall producer buffer utilization when deciding scale the writers.
9 tasks
34 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
We observe two cases in production cause current scale writer heuristics
not able to scale:
producer buffer is not overutilized.
buffer to be overutilized.
This commit tries to improve the situation by considering overall
producer buffer utilization when deciding scale the writers.