messages: fix second part of vttablet bug#3163
Merged
sougou merged 1 commit intovitessio:masterfrom Sep 13, 2017
sougou:messages
Merged
messages: fix second part of vttablet bug#3163sougou merged 1 commit intovitessio:masterfrom sougou:messages
sougou merged 1 commit intovitessio:masterfrom
sougou:messages
Conversation
BUG=65416566 When vttablet tries to send a new message to vtgate and receives EOF error, vttablet considers that as a real error and does an exponential back-off for retries. That means when vttablet sits idle for a long time (there are no new messages), and all clients reopened their MessageStream RPCs (which means a lot of dead RPCs due to point #1 above), then when new message arrives vttablet will receive a lot of EOF errors while it tries to send the message over dead RPCs. Due to exponential back-off real delivery of this message will be delayed significantly. If a send returns EOF, the sender does not postpone the messages. Instead, it turns on the messages pending flag and returns. This will cause the poller to be woken up by the send loop as soon as it gets through the cache.
Contributor
|
LGTM |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
BUG=65416566
When vttablet tries to send a new message to vtgate and receives EOF
error, vttablet considers that as a real error and does an exponential
back-off for retries. That means when vttablet sits idle for a long time
(there are no new messages), and all clients reopened their
MessageStream RPCs (which means a lot of dead RPCs due to point #1
above), then when new message arrives vttablet will receive a lot of EOF
errors while it tries to send the message over dead RPCs. Due to
exponential back-off real delivery of this message will be delayed
significantly.
If a send returns EOF, the sender does not postpone the messages.
Instead, it turns on the messages pending flag and returns. This
will cause the poller to be woken up by the send loop as soon as it
gets through the cache.