VStream API: allow keyspace-level heartbeats to be streamed by rohit-nayak-ps · Pull Request #16593 · vitessio/vitess

rohit-nayak-ps · 2024-08-13T21:21:14Z

Description

This PR adds an option to the VTGate VStream API to allow heartbeats created by vttablet's heartbeat writer to be streamed as regular FieldEvents/RowEvents for the sidecar _vt.heartbeat table.

flags := &vtgatepb.VStreamFlags{
	StreamKeyspaceHeartbeats: true,
}

Note that the vttablet heartbeats have to be enabled either through the throttler (which is on by default) or explicitly turning on the heartbeats using, for example, --heartbeat_enable, --heartbeat_interval 1s and --heartbeat_on_demand_duration 0

Related Issue(s)

Fixes: Feature Request: Binlog Timestamp Watermarking for VStream #16477

Checklist

"Backport to:" labels have been added if this change should be back-ported to release branches
If this change is to be back-ported to previous releases, a justification is included in the PR description
Tests were added or are not required
Did the new or modified tests pass consistently locally and on CI?
Documentation was added or is not required

Signed-off-by: Rohit Nayak <rohit@planetscale.com>

…l tables Signed-off-by: Rohit Nayak <rohit@planetscale.com>

Signed-off-by: Rohit Nayak <rohit@planetscale.com>

codecov · 2024-08-14T08:39:06Z

Codecov Report

Attention: Patch coverage is 90.12346% with 8 lines in your changes missing coverage. Please review.

Project coverage is 68.98%. Comparing base (cc68dd5) to head (8fb0240).
Report is 34 commits behind head on main.

Files	Patch %	Lines
go/vt/vttablet/tabletserver/vstreamer/vstreamer.go	94.33%	3 Missing ⚠️
go/vt/vtgate/vstream_manager.go	75.00%	2 Missing ⚠️
...et/tabletmanager/vreplication/replica_connector.go	0.00%	1 Missing ⚠️
go/vt/vttablet/tabletserver/binlog_watcher.go	0.00%	1 Missing ⚠️
go/vt/vttablet/tabletserver/tabletserver.go	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #16593      +/-   ##
==========================================
+ Coverage   68.85%   68.98%   +0.12%     
==========================================
  Files        1557     1562       +5     
  Lines      199891   200776     +885     
==========================================
+ Hits       137644   138505     +861     
- Misses      62247    62271      +24

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

…nes and the ones specified separately Signed-off-by: Rohit Nayak <rohit@planetscale.com>

Signed-off-by: Rohit Nayak <rohit@planetscale.com>

proto/binlogdata.proto

proto/vtgate.proto

go/vt/vtgate/vstream_manager.go

go/vt/vttablet/tabletserver/vstreamer/vstreamer.go

go/test/endtoend/vreplication/vstream_test.go

proto/binlogdata.proto

go/vt/vtgate/vstream_manager.go

mattlord · 2024-08-19T18:11:29Z

go/vt/vttablet/tabletserver/tabletserver.go

 		return err
 	}
-	return tsv.vstreamer.Stream(ctx, request.Position, request.TableLastPKs, request.Filter, throttlerapp.VStreamerName, send)
+	return tsv.vstreamer.Stream(ctx, request.Position, request.TableLastPKs, request.Filter, throttlerapp.VStreamerName, send, request.Options)


Is it worth adding vstream options to support a variadic set of options (0-N) like we recently did for lock options and (workflow) server options? Then we don't need to pass this as nil in so many places: https://github.com/vitessio/vitess/blob/main/go/vt/vtctl/workflow/server_options.go

I prefer using strong typing unless functionally contraindicated. Especially for a grpc API.

While updating all usages can be tedious, it also helps review the parts of code where that API is being called: #16593 (comment)

mattlord

LGTM! I only had some minor comments. Let me know what you think and I'll come back to this quickly.

go/vt/vttablet/tabletserver/vstreamer/helper_event_test.go

mattlord · 2024-08-19T18:37:30Z

go/vt/vttablet/tabletserver/vstreamer/vstreamer_test.go

-func TestInternalTables(t *testing.T) {
+func TestOnlineDDLTables(t *testing.T) {


This is a clue that it might be better to explicitly call these sidecar[db]_tables rather than internal_tables. Although they're all internal tables so it's also fine as-is IMO.

yeah, I was also vacillating between internal and sidecar(db). internal is not precise because we could also term online ddl shadow tables and other artifacts as internal. I will let @deepthi also weigh in before changing it everywhere.

This seems to have been resolved.

@deepthi, the question was a broader one. Currently we use options/proto attributes like InternalTables, is_internal. Should we change that to SidecarDBTables and is_sidecardb_table?

We could either continue calling it Internal (keeping it generic, allowing for future specification of other tables than SidecarDB tables, like, I don't know, shadow tables in OnlineDDL for example) or lock it down to SidecarDB tables using that in the object naming instead.

internal is fine for now. Renaming proto fields is not difficult if we wanted to change it in the future.

mattlord · 2024-08-19T18:38:28Z

go/vt/vttablet/tabletserver/vstreamer/vstreamer_test.go

 	runCases(t, filter, testcases, startPos, nil)
 }

+// TestInternalTables tests streaming of sidecar db tables.


This comment for the function name is another clue that it might be better/clearer to call them SidecarDBTables rather than InternalTables.

go/vt/vttablet/tabletserver/vstreamer/vstreamer_test.go

Signed-off-by: Rohit Nayak <rohit@planetscale.com>

deepthi · 2024-08-23T21:24:26Z

go/vt/vttablet/tabletserver/vstreamer/vstreamer_test.go

-func TestInternalTables(t *testing.T) {
+func TestOnlineDDLTables(t *testing.T) {


This seems to have been resolved.

go/vt/vttablet/tabletmanager/vreplication/external_connector.go

deepthi · 2024-08-23T21:35:28Z

go/vt/vttablet/tabletmanager/vreplication/replica_connector.go


 func (c *ReplicaConnector) VStream(ctx context.Context, startPos string, filter *binlogdatapb.Filter, send func([]*binlogdatapb.VEvent) error) error {
-	return c.vstreamer.Stream(ctx, startPos, nil, filter, throttlerapp.ReplicaConnectorName, send)
+	return c.vstreamer.Stream(ctx, startPos, nil, filter, throttlerapp.ReplicaConnectorName, send, nil)


Here again, based on how this is being used (only in PITR), we don't pass in the options.
Is it possible to replace usages of this connector with the tabletConnector?

Is it possible to replace usages of this connector with the tabletConnector?

I think so. Should be a separate PR though.

We should even think about deleting all the binlog server code in the existing PITR implementation. No one is using it.

go/vt/vttablet/tabletserver/vstreamer/vstreamer.go

Signed-off-by: Rohit Nayak <rohit@planetscale.com>

mattlord

Nice work, @rohit-nayak-ps ! ❤️

…#16593) Signed-off-by: Rohit Nayak <rohit@planetscale.com> Signed-off-by: Malcolm Akinje <makinje@slack-corp.com>

…#16593) Signed-off-by: Malcolm Akinje <makinje@slack-corp.com>

…#16593) (#620) * VStream API: allow keyspace-level heartbeats to be streamed (vitessio#16593) Signed-off-by: Malcolm Akinje <makinje@slack-corp.com> * `slack-19.0` backport v22 `vtorc` optimizations + stats, part 3 (#618) * Remove unused code in discovery queue creation (vitessio#17515) Signed-off-by: Manan Gupta <manan@planetscale.com> * vtorc: Cleanup unused code (vitessio#15508) Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com> * `vtorc`: cleanup discover queue, add concurrency flag (vitessio#17825) Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * `vtorc`: add tablets watched stats Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * fix missing merge conflict update Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * `vtorc`: skip unnecessary `inst.ReadTablet` in `logic.LockShard(...)` Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * `vtorc`: use `errgroup` in keyspace/shard discovery Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * fix import Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * fix ineffassign Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * missing import Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * `vtorc`: add stats for discovery workers Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * get count from backend Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * rm unused map Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> --------- Signed-off-by: Manan Gupta <manan@planetscale.com> Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com> Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> Co-authored-by: Manan Gupta <35839558+GuptaManan100@users.noreply.github.com> Co-authored-by: Dirkjan Bussink <d.bussink@gmail.com> * Bp pr 17558 pr 17858.slack19.0 (#615) * VReplication: Improve error handling in VTGate VStreams (vitessio#17558) Signed-off-by: Tom Thornton <thomaswilliamthornton@gmail.com> * Backport vitessio#17858 --------- Signed-off-by: Tom Thornton <thomaswilliamthornton@gmail.com> * `slack-19.0`: re-backport tweaks from vitessio#17911 (#621) * fix bug in reverse `if` Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * simplify Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * add `ReadTabletCountsByShard` test Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * use map of map Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * capitalize Cell Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * gofmt lint Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * fix plural in names Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> --------- Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> --------- Signed-off-by: Malcolm Akinje <makinje@slack-corp.com> Signed-off-by: Manan Gupta <manan@planetscale.com> Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com> Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> Signed-off-by: Tom Thornton <thomaswilliamthornton@gmail.com> Signed-off-by: Malcolm Akinje <malcolm.akinje@gmail.com> Co-authored-by: Tim Vaillancourt <tim@timvaillancourt.com> Co-authored-by: Manan Gupta <35839558+GuptaManan100@users.noreply.github.com> Co-authored-by: Dirkjan Bussink <d.bussink@gmail.com> Co-authored-by: Tom Thornton <thomaswilliamthornton@gmail.com>

…d Journal Events (#585) * VTGate VStream: Ensure reasonable delivery time for reshard journal event (vitessio#16639) Signed-off-by: Malcolm Akinje <malcolm.akinje@gmail.com> Signed-off-by: Malcolm Akinje <makinje@slack-corp.com> * Backport sqlparser patch for v15->v19 upgrade: 14763 Fix accepting bind variables in time related function calls (#590) * Fix accepting bind variables in time related function calls. (vitessio#14763) Signed-off-by: Manan Gupta <manan@planetscale.com> * fix test --------- Signed-off-by: Manan Gupta <manan@planetscale.com> Co-authored-by: Manan Gupta <35839558+GuptaManan100@users.noreply.github.com> * Upgrade vitess addons to 0.19.8 (#591) This upgrade allows us to control whether vtorc raises problems or not via an environment variable. Signed-off-by: Eduardo J. Ortega U. <5791035+ejortegau@users.noreply.github.com> * Use prefix in all vtorc check and recover logs (vitessio#17526) (#592) This is a backport of vitessio#17526 . Original PR description below: Description This is meant to make recovery actions more easily identified from the logs. See vitessio#17465 Signed-off-by: Eduardo J. Ortega U. <5791035+ejortegau@users.noreply.github.com> * `slack-19.0`: various backports for `vtorc`, part 2 (#596) * Ensure all topo read calls consider `--topo_read_concurrency` (vitessio#17276) Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * Revert "add keyrange support for vtorc clusters_to_watch (#457)" This reverts commit 45c2199. * [release-19.0] `vtorc`: require topo for `Healthy: true` in `/debug/health` (vitessio#17129) (vitessio#17351) Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> Signed-off-by: Manan Gupta <manan@planetscale.com> Co-authored-by: vitess-bot[bot] <108069721+vitess-bot[bot]@users.noreply.github.com> Co-authored-by: Tim Vaillancourt <tim@timvaillancourt.com> Co-authored-by: Manan Gupta <manan@planetscale.com> * `vtorc`: fetch all tablets from cells once + filter during refresh (vitessio#17388) Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * Support KeyRange in `--clusters_to_watch` flag (vitessio#17604) Signed-off-by: Manan Gupta <manan@planetscale.com> * missing func Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * Add api end point to print the current database state in VTOrc (vitessio#15485) Signed-off-by: Manan Gupta <manan@planetscale.com> --------- Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> Signed-off-by: Manan Gupta <manan@planetscale.com> Co-authored-by: vitess-bot[bot] <108069721+vitess-bot[bot]@users.noreply.github.com> Co-authored-by: Manan Gupta <manan@planetscale.com> Co-authored-by: Manan Gupta <35839558+GuptaManan100@users.noreply.github.com> * `slack-19.0`: `vtorc`: improve handling of partial cell topo results (#599) * `vtorc`: improve handling of partial cell topo results Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * add unit test Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * improve test Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * add comments Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * move sort to test Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * goimports Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> --------- Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * `slack-19.0`: skip tests that will fail on v15 downgrade testing (#605) Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * `slack-19.0`: Add stats for shards watched by VTOrc (#606) * Add stats for shards watched by VTOrc Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * Use len() in make --------- Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * Add `GetServerStatus` RPC to use in PRS (vitessio#16022) (#607) Signed-off-by: Manan Gupta <manan@planetscale.com> Co-authored-by: Manan Gupta <35839558+GuptaManan100@users.noreply.github.com> * backport/patch connection pool bug/perf fixes (#604) * [release-19.0] smartconnpool: do not allow connections to starve (vitessio#17675) (vitessio#17683) Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com> Co-authored-by: vitess-bot[bot] <108069721+vitess-bot[bot]@users.noreply.github.com> * smartconnpool: Better handling for idle expiration (vitessio#17756) Signed-off-by: Vicent Marti <vmg@strn.cat> --------- Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com> Signed-off-by: Vicent Marti <vmg@strn.cat> Co-authored-by: vitess-bot[bot] <108069721+vitess-bot[bot]@users.noreply.github.com> Co-authored-by: Vicent Martí <42793+vmg@users.noreply.github.com> Co-authored-by: Tim Vaillancourt <tim@timvaillancourt.com> * pool: reopen connection closed by idle timeout (vitessio#17818) (#609) Signed-off-by: Harshit Gangal <harshit@planetscale.com> Signed-off-by: Vicent Martí <42793+vmg@users.noreply.github.com> Co-authored-by: Harshit Gangal <harshit@planetscale.com> Co-authored-by: Vicent Martí <42793+vmg@users.noreply.github.com> * VReplication: Support excluding lagging tablets and use this in vstream manager (vitessio#17835) (#612) * `slack-19.0`: backport v22 VTOrc optimizations, part 2 (#613) * `vtorc`: remove duplicate instance read from backend (vitessio#17834) Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * `vtorc`: add index for `inst.ReadInstanceClusterAttributes` table scan Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> --------- Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * Add stats for shards watched by VTOrc, purge stale shards (vitessio#17815) (#616) * --consolidator-query-waiter-cap to set the max number of waiter for consolidated query (vitessio#17244) (#614) Signed-off-by: Jun Wang <jun.wang@demonware.net> Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> Co-authored-by: jwang <121262788+jwangace@users.noreply.github.com> Co-authored-by: Jun Wang <jun.wang@demonware.net> * `slack-19.0` backport v22 `vtorc` optimizations + stats, part 3 (#618) * Remove unused code in discovery queue creation (vitessio#17515) Signed-off-by: Manan Gupta <manan@planetscale.com> * vtorc: Cleanup unused code (vitessio#15508) Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com> * `vtorc`: cleanup discover queue, add concurrency flag (vitessio#17825) Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * `vtorc`: add tablets watched stats Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * fix missing merge conflict update Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * `vtorc`: skip unnecessary `inst.ReadTablet` in `logic.LockShard(...)` Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * `vtorc`: use `errgroup` in keyspace/shard discovery Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * fix import Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * fix ineffassign Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * missing import Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * `vtorc`: add stats for discovery workers Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * get count from backend Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * rm unused map Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> --------- Signed-off-by: Manan Gupta <manan@planetscale.com> Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com> Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> Co-authored-by: Manan Gupta <35839558+GuptaManan100@users.noreply.github.com> Co-authored-by: Dirkjan Bussink <d.bussink@gmail.com> * Bp pr 17558 pr 17858.slack19.0 (#615) * VReplication: Improve error handling in VTGate VStreams (vitessio#17558) Signed-off-by: Tom Thornton <thomaswilliamthornton@gmail.com> * Backport vitessio#17858 --------- Signed-off-by: Tom Thornton <thomaswilliamthornton@gmail.com> * `slack-19.0`: re-backport tweaks from vitessio#17911 (#621) * fix bug in reverse `if` Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * simplify Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * add `ReadTabletCountsByShard` test Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * use map of map Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * capitalize Cell Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * gofmt lint Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * fix plural in names Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> --------- Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * fix releasing the global read lock when mysqlshell backup fails (vitessio#17000) (#623) Signed-off-by: Renan Rangel <rrangel@slack-corp.com> * VStream API: allow keyspace-level heartbeats to be streamed (vitessio#16593) (#620) * VStream API: allow keyspace-level heartbeats to be streamed (vitessio#16593) Signed-off-by: Malcolm Akinje <makinje@slack-corp.com> * `slack-19.0` backport v22 `vtorc` optimizations + stats, part 3 (#618) * Remove unused code in discovery queue creation (vitessio#17515) Signed-off-by: Manan Gupta <manan@planetscale.com> * vtorc: Cleanup unused code (vitessio#15508) Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com> * `vtorc`: cleanup discover queue, add concurrency flag (vitessio#17825) Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * `vtorc`: add tablets watched stats Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * fix missing merge conflict update Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * `vtorc`: skip unnecessary `inst.ReadTablet` in `logic.LockShard(...)` Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * `vtorc`: use `errgroup` in keyspace/shard discovery Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * fix import Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * fix ineffassign Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * missing import Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * `vtorc`: add stats for discovery workers Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * get count from backend Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * rm unused map Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> --------- Signed-off-by: Manan Gupta <manan@planetscale.com> Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com> Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> Co-authored-by: Manan Gupta <35839558+GuptaManan100@users.noreply.github.com> Co-authored-by: Dirkjan Bussink <d.bussink@gmail.com> * Bp pr 17558 pr 17858.slack19.0 (#615) * VReplication: Improve error handling in VTGate VStreams (vitessio#17558) Signed-off-by: Tom Thornton <thomaswilliamthornton@gmail.com> * Backport vitessio#17858 --------- Signed-off-by: Tom Thornton <thomaswilliamthornton@gmail.com> * `slack-19.0`: re-backport tweaks from vitessio#17911 (#621) * fix bug in reverse `if` Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * simplify Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * add `ReadTabletCountsByShard` test Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * use map of map Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * capitalize Cell Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * gofmt lint Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * fix plural in names Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> --------- Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> --------- Signed-off-by: Malcolm Akinje <makinje@slack-corp.com> Signed-off-by: Manan Gupta <manan@planetscale.com> Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com> Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> Signed-off-by: Tom Thornton <thomaswilliamthornton@gmail.com> Signed-off-by: Malcolm Akinje <malcolm.akinje@gmail.com> Co-authored-by: Tim Vaillancourt <tim@timvaillancourt.com> Co-authored-by: Manan Gupta <35839558+GuptaManan100@users.noreply.github.com> Co-authored-by: Dirkjan Bussink <d.bussink@gmail.com> Co-authored-by: Tom Thornton <thomaswilliamthornton@gmail.com> * Increase health check channel buffer (vitessio#17821) (#625) Signed-off-by: Manan Gupta <manan@planetscale.com> Signed-off-by: Malcolm Akinje <makinje@slack-corp.com> Co-authored-by: Manan Gupta <35839558+GuptaManan100@users.noreply.github.com> * VStream: Allow for automatic resume after Reshard across VStreams (vitessio#15393) (#627) Signed-off-by: Tanjin Xu <tanjin.xu@slack-corp.com> Co-authored-by: Matt Lord <mattalord@gmail.com> --------- Signed-off-by: Malcolm Akinje <malcolm.akinje@gmail.com> Signed-off-by: Malcolm Akinje <makinje@slack-corp.com> Signed-off-by: Manan Gupta <manan@planetscale.com> Signed-off-by: Eduardo J. Ortega U. <5791035+ejortegau@users.noreply.github.com> Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com> Signed-off-by: Vicent Marti <vmg@strn.cat> Signed-off-by: Harshit Gangal <harshit@planetscale.com> Signed-off-by: Vicent Martí <42793+vmg@users.noreply.github.com> Signed-off-by: Jun Wang <jun.wang@demonware.net> Signed-off-by: Tom Thornton <thomaswilliamthornton@gmail.com> Signed-off-by: Renan Rangel <rrangel@slack-corp.com> Signed-off-by: Tanjin Xu <tanjin.xu@slack-corp.com> Co-authored-by: Tanjin Xu <109303790+tanjinx@users.noreply.github.com> Co-authored-by: Manan Gupta <35839558+GuptaManan100@users.noreply.github.com> Co-authored-by: Eduardo J. Ortega U. <5791035+ejortegau@users.noreply.github.com> Co-authored-by: Tim Vaillancourt <tim@timvaillancourt.com> Co-authored-by: vitess-bot[bot] <108069721+vitess-bot[bot]@users.noreply.github.com> Co-authored-by: Manan Gupta <manan@planetscale.com> Co-authored-by: Vicent Martí <42793+vmg@users.noreply.github.com> Co-authored-by: Harshit Gangal <harshit@planetscale.com> Co-authored-by: Tom Thornton <thomaswilliamthornton@gmail.com> Co-authored-by: jwang <121262788+jwangace@users.noreply.github.com> Co-authored-by: Jun Wang <jun.wang@demonware.net> Co-authored-by: Dirkjan Bussink <d.bussink@gmail.com> Co-authored-by: Renan Rangel <rvrangel@users.noreply.github.com> Co-authored-by: Matt Lord <mattalord@gmail.com>

github-actions bot added this to the v21.0.0 milestone Aug 13, 2024

rohit-nayak-ps added Type: Enhancement Logical improvement (somewhere between a bug and feature) Component: VReplication Component: Throttler Component: VTGate labels Aug 13, 2024

rohit-nayak-ps added 11 commits August 14, 2024 10:17

Add VStream flag to enable heartbeats

78687fc

Signed-off-by: Rohit Nayak <rohit@planetscale.com>

Add e2e test

657ccca

Signed-off-by: Rohit Nayak <rohit@planetscale.com>

Always send heartbeat table events on

1085d56

Signed-off-by: Rohit Nayak <rohit@planetscale.com>

Add IsInternal to FIELD/ROW events for sidecar tables

2af77cc

Signed-off-by: Rohit Nayak <rohit@planetscale.com>

Pass options through to VStreamer

0a24b85

Signed-off-by: Rohit Nayak <rohit@planetscale.com>

Fix unit tests for new Stream signature

6d55bb7

Signed-off-by: Rohit Nayak <rohit@planetscale.com>

Minor refactor

f00e37c

Signed-off-by: Rohit Nayak <rohit@planetscale.com>

Allow streaming of any internal tables. Add unit test. Update e2e test

25c92ef

Signed-off-by: Rohit Nayak <rohit@planetscale.com>

Don't error out while dropping tables in defer to account for interna…

fa40b47

…l tables Signed-off-by: Rohit Nayak <rohit@planetscale.com>

Only expose keyspace heartbeats in VTGate API

d516f8b

Signed-off-by: Rohit Nayak <rohit@planetscale.com>

Fix vstream call for new signature in vdiff framework

731799a

Signed-off-by: Rohit Nayak <rohit@planetscale.com>

rohit-nayak-ps force-pushed the rohit/vstream-vt-heartbeat branch from 68863d4 to 731799a Compare August 14, 2024 08:17

rohit-nayak-ps added 2 commits August 14, 2024 11:47

If streaming from a sidecar stream all tables, not just the special o…

22fade6

…nes and the ones specified separately Signed-off-by: Rohit Nayak <rohit@planetscale.com>

Self-review

45edbb1

Signed-off-by: Rohit Nayak <rohit@planetscale.com>

rohit-nayak-ps mentioned this pull request Aug 14, 2024

Feature Request: Binlog Timestamp Watermarking for VStream #16477

Closed

rohit-nayak-ps added 2 commits August 16, 2024 08:15

Self-review: e2e test case

ad21a8e

Signed-off-by: Rohit Nayak <rohit@planetscale.com>

Self-review

5221811

Signed-off-by: Rohit Nayak <rohit@planetscale.com>

rohit-nayak-ps marked this pull request as ready for review August 16, 2024 08:41

rohit-nayak-ps requested review from deepthi, derekperkins, harshit-gangal, mattlord, shlomi-noach and systay as code owners August 16, 2024 08:41

rohit-nayak-ps requested review from GuptaManan100, ajm188 and timvaillancourt as code owners August 16, 2024 08:41

rohit-nayak-ps mentioned this pull request Aug 18, 2024

VReplication: make flags workflow-specific and dynamically changeable #16583

Merged

5 tasks

harshit-gangal reviewed Aug 19, 2024

View reviewed changes

proto/binlogdata.proto Show resolved Hide resolved

harshit-gangal reviewed Aug 19, 2024

View reviewed changes

proto/vtgate.proto Show resolved Hide resolved

harshit-gangal reviewed Aug 19, 2024

View reviewed changes

go/vt/vtgate/vstream_manager.go Outdated Show resolved Hide resolved

harshit-gangal reviewed Aug 19, 2024

View reviewed changes

go/vt/vttablet/tabletserver/vstreamer/vstreamer.go Outdated Show resolved Hide resolved

mattlord reviewed Aug 19, 2024

View reviewed changes

Address review comments

06e7b17

Signed-off-by: Rohit Nayak <rohit@planetscale.com>

deepthi reviewed Aug 23, 2024

View reviewed changes

rohit-nayak-ps added 2 commits August 24, 2024 19:26

Address review comments

2f63e6a

Signed-off-by: Rohit Nayak <rohit@planetscale.com>

Address review comments: change IsInternal to IsInternalTable

8fb0240

Signed-off-by: Rohit Nayak <rohit@planetscale.com>

mattlord approved these changes Aug 26, 2024

View reviewed changes

deepthi approved these changes Aug 27, 2024

View reviewed changes

deepthi merged commit bc6fb1c into vitessio:main Aug 27, 2024

deepthi deleted the rohit/vstream-vt-heartbeat branch August 27, 2024 16:09

makinje16 mentioned this pull request Jan 16, 2025

VStream API [Backport]: Allow Keyspace-Level Heartbeats to be Streamed (#16593) slackhq/vitess#586

Closed

4 tasks

makinje16 pushed a commit to slackhq/vitess that referenced this pull request Mar 7, 2025

VStream API: allow keyspace-level heartbeats to be streamed (vitessio…

bd4696d

…#16593) Signed-off-by: Rohit Nayak <rohit@planetscale.com> Signed-off-by: Malcolm Akinje <makinje@slack-corp.com>

tanjinx mentioned this pull request Mar 12, 2025

VStream API: allow keyspace-level heartbeats to be streamed (#16593) slackhq/vitess#620

Merged

5 tasks

makinje16 added a commit to slackhq/vitess that referenced this pull request Mar 13, 2025

VStream API: allow keyspace-level heartbeats to be streamed (vitessio…

936dffe

…#16593) Signed-off-by: Malcolm Akinje <makinje@slack-corp.com>

		func TestInternalTables(t *testing.T) {
		func TestOnlineDDLTables(t *testing.T) {

Conversation

rohit-nayak-ps commented Aug 13, 2024 • edited by mattlord Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Issue(s)

Checklist

Uh oh!

codecov bot commented Aug 14, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mattlord left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rohit-nayak-ps Aug 24, 2024 • edited by deepthi Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mattlord left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

rohit-nayak-ps commented Aug 13, 2024 •

edited by mattlord

Loading

codecov bot commented Aug 14, 2024 •

edited

Loading

rohit-nayak-ps Aug 24, 2024 •

edited by deepthi

Loading