resharding: auto-reverse replication#4262
Conversation
MigrateServedTypes for master automatically does the prep work for reversing replication. If the flag is specified, it reverses the direction thereby allowing you to migrate back to the original masters. If the flag was not originally specified, then it only does the prep work, without actually starting the replication. You can later reissue the Migrate command, which will then start the vreplication. Signed-off-by: Sugu Sougoumarane <ssougou@gmail.com>
|
WHOA! |
| reverse := subFlags.Bool("reverse", false, "Moves the served tablet type backward instead of forward. Use in case of trouble") | ||
| skipReFreshState := subFlags.Bool("skip-refresh-state", false, "Skips refreshing the state of the source tablets after the migration, meaning that the refresh will need to be done manually, replica and rdonly only)") | ||
| filteredReplicationWaitTime := subFlags.Duration("filtered_replication_wait_time", 30*time.Second, "Specifies the maximum time to wait, in seconds, for filtered replication to catch up on master migrations") | ||
| reverseReplication := subFlags.Bool("reverse_replication", false, "For master migration, enabling this flag reverses replication allowing which allows you to rollback") |
|
|
||
| // Phase 2 | ||
| // Always setup reverse replication. We'll start it later if reverseReplication was specified. | ||
| // This will allow someone to reverse the replication later if they change their mind. |
There was a problem hiding this comment.
Ah so even if you don't pass the reverse_replication flag you can still start vreplication and revert the migration?
There was a problem hiding this comment.
In fact, you should't start vreplication yourself. You should just reissue MigrateServedTypes with -reverse_replication.
|
@demmer @tirsen Need your thoughts on something @rafael and I have been debating. The current implementation of this feature is as follows:
@rafael proposes we change this to:
The current approach: @rafael proposal: Let us know if you have any preferences. |
|
I think the safest option should be the default but this is a somewhat unproven option so maybe as a first step make it not the default and then we make it the default once it's been well tested in the field. |
|
Seems crazy not to save the metadata though. |
|
Clarified with @tirsen: auto-reverse should not happen unless requested, but metadata should be created irrespective of the flag. This is the current implementation. |
|
My preference would be somewhere in between... Could we unconditionally save the master position metadata that you would need to start reverse replication, but don't set up the replication itself unless requested? It could be a new workflow type of "SavedReverseReplicationPos" or some such which would never actually be used to start the replication, but from which we could get the positions if needed. |
|
We don't currently use workflows so we wouldn't benefit from saving stuff there. In my opinion safest option should always be the default. Based on that philosophy by default we should even start the reverse vreplication. That said, this is still a somewhat unproven feature so might need to see some more field testing first. As a first step setting up the vreplication but not starting it seems like a safe low risk option. |
|
Per discussion so far with @demmer, he's leaning towards @rafael's approach, which is to not setup reverse replication if it was not requested. It's a tie. But since I'm not a user, I'll count my vote as the weakest. I'm still trying to make a case for this. However, if I don't succeed, the decision will be to not provide the possibility of later reversing replication if it was not requested at the time of issuing the Migrate. If so, the third option suggested by @demmer can be considered for the future. |
|
For posterity, which approach did we end up merging? (Too lazy to read the code.) |
3 similar comments
|
For posterity, which approach did we end up merging? (Too lazy to read the code.) |
|
For posterity, which approach did we end up merging? (Too lazy to read the code.) |
|
For posterity, which approach did we end up merging? (Too lazy to read the code.) |
|
Always save metadata. Initiate auto-reverse only if requested. |
4 participants
MigrateServedTypes for master automatically does the prep work for
reversing replication. If the flag is specified, it reverses the
direction thereby allowing you to migrate back to the original
masters.
If the flag was not originally specified, then it only does the prep
work, without actually starting the replication. You can later
reissue the Migrate command, which will then start the vreplication.
Signed-off-by: Sugu Sougoumarane ssougou@gmail.com