Manage MySQL Replication Status States Properly#9853
Merged
deepthi merged 9 commits intovitessio:mainfrom Mar 11, 2022
Merged
Conversation
…failed Signed-off-by: Matt Lord <mattalord@gmail.com>
…ationStatus Signed-off-by: Matt Lord <mattalord@gmail.com>
Signed-off-by: Matt Lord <mattalord@gmail.com>
Signed-off-by: Matt Lord <mattalord@gmail.com>
Signed-off-by: Matt Lord <mattalord@gmail.com>
And make function names nicer (removing double context) And add some helper functions Signed-off-by: Matt Lord <mattalord@gmail.com>
Signed-off-by: Matt Lord <mattalord@gmail.com>
Signed-off-by: Matt Lord <mattalord@gmail.com>
Update vtadmin web protobufs Signed-off-by: Matt Lord <mattalord@gmail.com>
3 tasks
Member
Author
|
@deepthi, @GuptaManan100, and @shlomi-noach: I apologize for the review hassle but unfortunately creating a new PR was the only way forward (see here for details). In the process, I consolidated the changes since the last round of reviews — focused on safely modifying the protobuf to fit the new model — into a single commit: 65226ad Thanks in advance! 🙇♂️ |
mattlord
requested review from
ajm188,
doeg,
harshit-gangal,
notfelineit,
rohit-nayak-ps and
systay
as code owners
mattlord
removed request for
ajm188,
doeg,
harshit-gangal,
rohit-nayak-ps and
systay
GuptaManan100
approved these changes
Mar 10, 2022
Contributor
GuptaManan100
left a comment
GuptaManan100
left a comment
There was a problem hiding this comment.
I feel sad that you had to jump through so many hoops for getting this in 😞
deepthi
approved these changes
Mar 11, 2022
This was referenced Mar 25, 2022
mattlord
added a commit
to planetscale/vitess
that referenced
this pull request
Apr 23, 2022
As release-13.0 does not have this: vitessio#9853 Signed-off-by: Matt Lord <mattalord@gmail.com>
mattlord
added a commit
that referenced
this pull request
Apr 24, 2022
…ded (#10123) * Only start SQL thread temporarily to WaitForPosition if needed (#10104) After #9512 we always attempted to start the replication SQL_Thread(s) when waiting for a given position. The problem with this, however, is that if the SQL_Thread is running but the IO_Thread is not then the tablet repair does not try and start replication on a replica tablet. So in certain states such as when initializing a shard, replication may end up in a non-healthy state and never be repaired. This changes the behavior so that: 1. We only attempt to start the SQL_Thread(s) if it's not already running 2. If we explicitly start the SQL_Thread(s) then we also explicitly reset it to what it was (stopped) as we exit the call Because the caller should be/have a TabletManager which has a mutex, this should ensure that the replication manager calls are serialized and because we are resetting the replication state after mutating it, everything should work as it did before #9512 with the exception being that when waiting we ensure that the replica at least has the possibility of catching up. Signed-off-by: Matt Lord <mattalord@gmail.com> * Use older replication status interface As release-13.0 does not have this: #9853 Signed-off-by: Matt Lord <mattalord@gmail.com>
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
We considered the
Slave_IO_Runningstate ofConnectingas equivalent toRunningin the replication status results that we get from MySQL. I'm assuming this was done to avoid flapping on low traffic systems due to the-slave_net_timeoutreconnects and avoid doing a tablet repair when replication was healthy (with this PR we can distinguish running from healthy states and take differing actions depending on what we want).After #9308 we properly estimate the replica lag when MySQL is telling us that it does not know, meaning it returns a
NULLvalue for theSeconds_Behind_Masterfield. But in some cases — e.g. it appears to happen when attempting the first connection to the source — MySQL will report aSeconds_Behind_Mastervalue of0(meaning fully caught up and no lag) even when it is not connected to its replication source and has failed to reconnect — meaning that this is not a simple reconnect for any reason (e.g.-slave_net_timeout), but a reconnect with one or more failures/errors. This PR handles this latter case within Vitess by continuing to treatConnectingas equivalent toRunningviaReplicationStatus.Healthy()— to prevent the noted flapping and errant/unnecessary tablet repairs — unless we had an IO error the last time we tried to reconnect to the replication source.Related Issue(s)
Checklist