Skip to content

Improvements to workflows v2#4157

Merged
sougou merged 21 commits intovitessio:masterfrom
tinyspeck:improvements-to-workflows-v2
Sep 1, 2018
Merged

Improvements to workflows v2#4157
sougou merged 21 commits intovitessio:masterfrom
tinyspeck:improvements-to-workflows-v2

Conversation

@rafael
Copy link
Copy Markdown
Member

@rafael rafael commented Aug 25, 2018

Desc

The following PR handles all the issues described in: #4117

I think I still need to update some tests, but wanted to open it so we can start taking a look at the overall changes.

It looks big but it's mostly cosmetic changes. Notable changes are:

  • Updates existent HorizontalResharding workflow to be explicit about the source and destination shards to be split/merged.
  • Extends enable approvals in HorizontalResharding to have more granularity. After this change, users can decide explicitly which phases they want to require enable approvals.
  • Adds a meta workflow that create an horizontal resharding workflow for all the available overlapping shards.
  • Adds create_time to workflow protobuf so we can sort deterministically.

Example

The following is an excerpt of an horizontal resharding from 2 to 4 shards. Only the migrate serving types require approval and the generated workflows are started automatically.

It's a bit hard to see the details, but you can see that now it creates two workflows, one for each shard that is being split spilt.

ezgif com-video-to-gif

Rafael Chacon added 12 commits August 16, 2018 13:58
… shards

Signed-off-by: Rafael Chacon <rafael@slack-corp.com>
Signed-off-by: Rafael Chacon <rafael@slack-corp.com>
Signed-off-by: Rafael Chacon <rafael@slack-corp.com>
…ocess

Signed-off-by: Rafael Chacon <rafael@slack-corp.com>
Signed-off-by: Rafael Chacon <rafael@slack-corp.com>
Signed-off-by: Rafael Chacon <rafael@slack-corp.com>
Signed-off-by: Rafael Chacon <rafael@slack-corp.com>
Signed-off-by: Rafael Chacon <rafael@slack-corp.com>
Signed-off-by: Rafael Chacon <rafael@slack-corp.com>
* This change allows us to sort workflows in a deterministic way.

Signed-off-by: Rafael Chacon <rafael@slack-corp.com>
Signed-off-by: Rafael Chacon <rafael@slack-corp.com>
@rafael rafael requested a review from demmer August 25, 2018 00:59
Copy link
Copy Markdown
Contributor

@jvaidya jvaidya left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good to me on the whole and I think this will be a big improvement for resharding workflow when you are resharding a large number of shards. One question: I do not see you validating the sourceShards and targetShards input parameters. In the case where a human is putting together a SplitDiff or a SplitClone commandline, it would be good to validate these.

@rafael
Copy link
Copy Markdown
Member Author

rafael commented Aug 27, 2018

@jvaidya oh good catch. Will add that validation.

Rafael Chacon added 6 commits August 27, 2018 10:27
Signed-off-by: Rafael Chacon <rafael@slack-corp.com>
* Parallel runner should be a generic component of workflows. They don't need to
  be part of resharding. This way other workflows can benefit from it.
* Add tests for horizontal resharding workflow creator.

Signed-off-by: Rafael Chacon <rafael@slack-corp.com>
Signed-off-by: Rafael Chacon <rafael@slack-corp.com>
Signed-off-by: Rafael Chacon <rafael@slack-corp.com>
Signed-off-by: Rafael Chacon <rafael@slack-corp.com>
Signed-off-by: Rafael Chacon <rafael@slack-corp.com>
Copy link
Copy Markdown
Contributor

@jvaidya jvaidya left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks for tackling this.

@rafael rafael changed the title WIP: Improvements to workflows v2 Improvements to workflows v2 Aug 28, 2018
Rafael Chacon added 3 commits August 27, 2018 19:41
Signed-off-by: Rafael Chacon <rafael@slack-corp.com>
Signed-off-by: Rafael Chacon <rafael@slack-corp.com>
Signed-off-by: Rafael Chacon <rafael@slack-corp.com>
@rafael
Copy link
Copy Markdown
Member Author

rafael commented Aug 30, 2018

I tested this in our dev environment. Created a keyspace with 8 shards and split to 16. The process was super smooth.

Copy link
Copy Markdown
Contributor

@sougou sougou left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving by proxy of @jvaidya

@sougou sougou merged commit fca9d15 into vitessio:master Sep 1, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants