vstreamer to throttle on source endpoint by shlomi-noach · Pull Request #7324 · vitessio/vitess

shlomi-noach · 2021-01-20T05:35:48Z

Description

VReplication works by hooking to a source tablet, pulling data as a replica as well as scanning tables, then to be applied on a target tablet.

On source, VReplication (via vstreamer) can cause a high load by aggressively pulling data, such that:

if source is a replica, it can start lagging
if source is a primary, it becomes sluggish

With this PR, vstreamer throttles based on check-self, introduced in #7319 , which is a throttle endpoint that tests the health of a specific tablet, the source tablet in our scenario. Right now, as described in #7319, whether replica or primary, we test for replication lag, which is an actual indicator on primary, too.

Related Issue(s)

Tracking issue: #7362

This PR extends and depends on #7319

Checklist

Should this PR be backported?
Tests were added or are not required
Documentation was added or is not required

Deployment Notes

Impacted Areas in Vitess

Components that this PR will affect:

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

…amer pulling data from source Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

…le() function Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

…rapping events channel with throttledEvents, which only pulls data from events if not throttled Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

shlomi-noach · 2021-01-20T17:50:33Z

@rohit-nayak-ps please see my recent changes. On vstreamer.go, I only throttle on the events channel, so that I'm not blocking venets and ctx.Done() and timer.C.

Otherwise, added throttling in rowstreamer and resultstreamer.

shlomi-noach · 2021-01-20T17:51:08Z

Next up, I'l create an endtoend test

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

…e-source-check-self Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

… keyword Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

shlomi-noach · 2021-01-21T09:46:38Z

I've moved rowstreamer and resultstreamer throttling inside their main loops, where throttling is more granular. In my understanding, where we initially placed throttling (at beginning of Stream() function) was a one time check at the point of initially setting up the streaming.

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

…e-source-check-self Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

shlomi-noach · 2021-01-24T11:08:47Z

Ready for review! Tests are now good.

Reminder that this PR extends #7319, so it's best to first merge #7319

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

rohit-nayak-ps · 2021-01-25T21:01:15Z

The new workflow yaml file for the e2e test should be created by updating the test/ci_workflow_gen.go codegen script and running it. Otherwise any updates to the cluster template file will not reflect in this yaml.

rohit-nayak-ps

looks great!
test script / e2e workflow yaml needs to be updated as per comment.

shlomi-noach · 2021-01-26T07:08:43Z

The new workflow yaml file for the e2e test should be created by updating the test/ci_workflow_gen.go codegen script

Oh, I completely missed this!

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

shlomi-noach · 2021-01-26T08:32:44Z

because this PR introduced a named endtoend shard (vreplication_basic), I took it to now normalize all shards to be strings, not numbers. So, for example, shard 22 is now shard "22".

shlomi-noach added 3 commits January 19, 2021 16:13

VStreamer throttles on source based on check-self

492ac3c

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

vstreamer.Engine now gets lagThrottler reference, to be used by vstre…

b2ee511

…amer pulling data from source Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

removed debug messages

2907746

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

shlomi-noach requested a review from rohit-nayak-ps January 20, 2021 05:44

shlomi-noach added 2 commits January 20, 2021 15:49

Engine owns throttling. Streamers of all types cal on Engine's thrott…

0eea641

…le() function Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

only throttle the events channel, not other channels. We do this by w…

f99fa1b

…rapping events channel with throttledEvents, which only pulls data from events if not throttled Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

shlomi-noach added 16 commits January 20, 2021 22:12

name of vreplication_basic workflow

542fb91

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

name of workflow file

dd14a08

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

vreplication_basic on its own shard

1e781d4

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

API endpoint for throttle-app and unthrottle-app

138bfd5

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

simplify

603838c

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

rename

5239392

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

test materialization with source throttling

67b5889

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

some comments, a minor Sleep

209e96e

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

vreplication_basic moved to a new shard

9b3e65a

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

Merge remote-tracking branch 'upstream/master' into vstreamer-throttl…

694e1f1

…e-source-check-self Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

sub test sections

d101493

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

enable throttler in vreplication test tablets

4bf07d3

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

fix shard name

7fa0763

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

exit endless loop

78cb68b

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

fix replication lag query: apparently 'lag' is now introduced asa new…

ace94f6

… keyword Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

more granular throttling

498ccf3

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

shlomi-noach added 5 commits January 21, 2021 13:16

fix vstreamer app name in test, plus validate it

ac494c2

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

fix throttler path

6d6e06b

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

minor refactor

e4a00b3

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

update api path

2c32c91

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

throttle on source tablets

f8822eb

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

shlomi-noach added 3 commits January 24, 2021 11:44

Merge remote-tracking branch 'upstream/master' into vstreamer-throttl…

5e24918

…e-source-check-self Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

fixed dependencies in tabletserver

dcb6915

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

comment

166d939

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

shlomi-noach changed the title ~~WIP: vstreamer to throttle on source endpoint~~ vstreamer to throttle on source endpoint Jan 24, 2021

shlomi-noach marked this pull request as ready for review January 24, 2021 11:08

shlomi-noach requested review from harshit-gangal, sougou and systay as code owners January 24, 2021 11:08

This was referenced Jan 24, 2021

Tablet throttle: support "/throttle/check-self" available on all tablets #7319

Merged

VReplication: throttle on target tablet #7364

Merged

shlomi-noach added 3 commits January 25, 2021 13:39

support 'TickNow()'

0bc14fa

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

speed up of throttler refresh tick upon opening and upon becoming leader

db121c1

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

restoring lost endtoend test(24). Whoops

bb328fa

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

rohit-nayak-ps approved these changes Jan 25, 2021

View reviewed changes

shlomi-noach added 4 commits January 26, 2021 09:36

merge master, resolve conflicts

13ca3db

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

workflow endtoend tests: named shards rather than numbered shards

2b4c5a2

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

convert shard numbers to strings

7f44ebc

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

shard -1 --> shard ""

22088b4

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

shlomi-noach mentioned this pull request Jan 26, 2021

Github Actions CI Builds: convert matrix strategy for unit and cluster tests to individual tests #7258

Merged

7 tasks

shlomi-noach merged commit fbb4300 into vitessio:master Jan 26, 2021

shlomi-noach deleted the vstreamer-throttle-source-check-self branch January 26, 2021 09:57

shlomi-noach mentioned this pull request Jan 27, 2021

Documenting VReplication throttling vitessio/website#689

Merged

askdba added Component: VReplication P3 labels Feb 8, 2021

askdba added this to the v10.0 milestone Feb 8, 2021

ajm188 mentioned this pull request Jul 13, 2021

slack vitess v10.pre tinyspeck/vitess#228

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

vstreamer to throttle on source endpoint#7324

vstreamer to throttle on source endpoint#7324
shlomi-noach merged 43 commits intovitessio:masterfrom
planetscale:vstreamer-throttle-source-check-self

shlomi-noach commented Jan 20, 2021 •

edited

Loading

Uh oh!

shlomi-noach commented Jan 20, 2021

Uh oh!

shlomi-noach commented Jan 20, 2021

Uh oh!

shlomi-noach commented Jan 21, 2021

Uh oh!

shlomi-noach commented Jan 24, 2021

Uh oh!

rohit-nayak-ps commented Jan 25, 2021 •

edited

Loading

Uh oh!

rohit-nayak-ps left a comment

Uh oh!

shlomi-noach commented Jan 26, 2021

Uh oh!

shlomi-noach commented Jan 26, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

shlomi-noach commented Jan 20, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Issue(s)

Checklist

Deployment Notes

Impacted Areas in Vitess

Uh oh!

shlomi-noach commented Jan 20, 2021

Uh oh!

shlomi-noach commented Jan 20, 2021

Uh oh!

shlomi-noach commented Jan 21, 2021

Uh oh!

shlomi-noach commented Jan 24, 2021

Uh oh!

rohit-nayak-ps commented Jan 25, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rohit-nayak-ps left a comment

Choose a reason for hiding this comment

Uh oh!

shlomi-noach commented Jan 26, 2021

Uh oh!

shlomi-noach commented Jan 26, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

shlomi-noach commented Jan 20, 2021 •

edited

Loading

rohit-nayak-ps commented Jan 25, 2021 •

edited

Loading