Add more vstream metrics for vstream manager by twthorn · Pull Request #17858 · vitessio/vitess

twthorn · 2025-02-24T22:56:27Z

Description

Add more metrics for vstreams on vtgates.

Inspired partially by the tablet vstream metrics. Let me know if you think I should include any others. Once we are decided on the metrics list, I will put out docs PR.

Related Issue(s)

Fixes: Feature Request: VTGates should report more metrics on vstreams #17859

Checklist

"Backport to:" labels have been added if this change should be back-ported to release branches
If this change is to be back-ported to previous releases, a justification is included in the PR description
Tests were added or are not required
Did the new or modified tests pass consistently locally and on CI?
Documentation: Document new metrics for vtgate vstream website#1948

vitess-bot · 2025-02-24T22:56:30Z

Signed-off-by: twthorn <thomaswilliamthornton@gmail.com>

codecov · 2025-02-25T17:49:43Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 67.47%. Comparing base (81ce29c) to head (239367b).
Report is 20 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff            @@
##             main   #17858    +/-   ##
========================================
  Coverage   67.46%   67.47%            
========================================
  Files        1593     1594     +1     
  Lines      258885   259089   +204     
========================================
+ Hits       174652   174814   +162     
- Misses      84233    84275    +42

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

mattlord

Thank you for working on this, @twthorn ! ❤️ I think that we should work out what we want these metrics to represent, and then we can refine from there. Hopefully all of my comments make sense?

go/vt/vtgate/vstream_manager.go

mattlord · 2025-02-25T21:54:19Z

go/vt/vtgate/vstream_manager.go

+		labels := []string{sgtid.Keyspace, sgtid.Shard, vs.tabletType.String()}
+		vs.vsm.vstreamsEndedWithErrors.Add(labels, 0)


I had assumed that this was a count of vtgate vstreams rather than tablet streams. Is this really supposed to be a count of tablet streams (one per shard, per vtgate vstream)?

I believe the most useful for observability is to have per shard per vtgate stream. For example, with the lag metric, if the vstream has multiple shards, we cannot accurately report lag (do we do the max, min, avg, etc.).

I believe the same is true

errors - it saves operator the time to find in a log which stream has error

active streams - if the vtgate is operating slow the vstream count is not enough granularity if those vstreams have hundreds of shards, we want to know how many shards this vtgate is handling

events streamed - help operator detect which shard has disproportionately high load

I agree per tablet is maybe too granular, but per-shard would be very useful. I am going to update PR based on this

OK, makes sense. Thanks!

go/vt/vtgate/vstream_manager.go

Signed-off-by: twthorn <thomaswilliamthornton@gmail.com>

mattlord

LGTM! I only had some minor comments and suggestions. Please let me know what you think.

Thanks again, @twthorn !

mattlord · 2025-02-27T15:19:42Z

go/vt/vtgate/vstream_manager.go

 		defer vs.wg.Done()
+
+		labelValues := []string{sgtid.Keyspace, sgtid.Shard, vs.tabletType.String()}
+		vs.vsm.vstreamsEndedWithErrors.Add(labelValues, 0)


I don't see any value in adding 0. Am I missing something?

This is to initialize the counter to zero for this keyspace/shard/tablet type. This is a best practice. Metrics are hard to work with if they only sometimes exist. This is also seen in the unit test for metrics that we can't assert that there were zero errors for a vstream that worked fine (the metric is simply missing)

I changed to Reset to hopefully make it more clear, as Add zero does appear to be not useful

Is Reset what you really want? It's a counter and not a gauge. I assumed that it was meant to be a counter that spanned the life of the vtgate as the description is "Number of vstreams that ended with errors".

Ah yes good point, this could be called multiple times. So we will just keep with Add zero.

go/vt/vtgate/vstream_manager_test.go

go/vt/vtgate/vstream_manager.go

Signed-off-by: twthorn <thomaswilliamthornton@gmail.com>

mattlord

Nice work on this, @twthorn ! ❤️ Much appreciated.

For docs, we should add these new metrics to the v22 docs here: https://vitess.io/docs/22.0/reference/vreplication/metrics/#vtgate-metrics

Do you mind opening a website/docs PR for that as well? I can help as needed.

twthorn · 2025-02-27T22:46:01Z

Sure thing, opened here vitessio/website#1948

* VReplication: Improve error handling in VTGate VStreams (vitessio#17558) Signed-off-by: Tom Thornton <thomaswilliamthornton@gmail.com> * Backport vitessio#17858 --------- Signed-off-by: Tom Thornton <thomaswilliamthornton@gmail.com>

…#16593) (#620) * VStream API: allow keyspace-level heartbeats to be streamed (vitessio#16593) Signed-off-by: Malcolm Akinje <makinje@slack-corp.com> * `slack-19.0` backport v22 `vtorc` optimizations + stats, part 3 (#618) * Remove unused code in discovery queue creation (vitessio#17515) Signed-off-by: Manan Gupta <manan@planetscale.com> * vtorc: Cleanup unused code (vitessio#15508) Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com> * `vtorc`: cleanup discover queue, add concurrency flag (vitessio#17825) Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * `vtorc`: add tablets watched stats Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * fix missing merge conflict update Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * `vtorc`: skip unnecessary `inst.ReadTablet` in `logic.LockShard(...)` Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * `vtorc`: use `errgroup` in keyspace/shard discovery Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * fix import Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * fix ineffassign Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * missing import Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * `vtorc`: add stats for discovery workers Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * get count from backend Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * rm unused map Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> --------- Signed-off-by: Manan Gupta <manan@planetscale.com> Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com> Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> Co-authored-by: Manan Gupta <35839558+GuptaManan100@users.noreply.github.com> Co-authored-by: Dirkjan Bussink <d.bussink@gmail.com> * Bp pr 17558 pr 17858.slack19.0 (#615) * VReplication: Improve error handling in VTGate VStreams (vitessio#17558) Signed-off-by: Tom Thornton <thomaswilliamthornton@gmail.com> * Backport vitessio#17858 --------- Signed-off-by: Tom Thornton <thomaswilliamthornton@gmail.com> * `slack-19.0`: re-backport tweaks from vitessio#17911 (#621) * fix bug in reverse `if` Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * simplify Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * add `ReadTabletCountsByShard` test Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * use map of map Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * capitalize Cell Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * gofmt lint Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * fix plural in names Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> --------- Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> --------- Signed-off-by: Malcolm Akinje <makinje@slack-corp.com> Signed-off-by: Manan Gupta <manan@planetscale.com> Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com> Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> Signed-off-by: Tom Thornton <thomaswilliamthornton@gmail.com> Signed-off-by: Malcolm Akinje <malcolm.akinje@gmail.com> Co-authored-by: Tim Vaillancourt <tim@timvaillancourt.com> Co-authored-by: Manan Gupta <35839558+GuptaManan100@users.noreply.github.com> Co-authored-by: Dirkjan Bussink <d.bussink@gmail.com> Co-authored-by: Tom Thornton <thomaswilliamthornton@gmail.com>

* VReplication: Improve error handling in VTGate VStreams (vitessio#17558) Signed-off-by: Tom Thornton <thomaswilliamthornton@gmail.com> * Backport vitessio#17858 --------- Signed-off-by: Tom Thornton <thomaswilliamthornton@gmail.com>

…#16593) (#620) * VStream API: allow keyspace-level heartbeats to be streamed (vitessio#16593) Signed-off-by: Malcolm Akinje <makinje@slack-corp.com> * `slack-19.0` backport v22 `vtorc` optimizations + stats, part 3 (#618) * Remove unused code in discovery queue creation (vitessio#17515) Signed-off-by: Manan Gupta <manan@planetscale.com> * vtorc: Cleanup unused code (vitessio#15508) Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com> * `vtorc`: cleanup discover queue, add concurrency flag (vitessio#17825) Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * `vtorc`: add tablets watched stats Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * fix missing merge conflict update Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * `vtorc`: skip unnecessary `inst.ReadTablet` in `logic.LockShard(...)` Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * `vtorc`: use `errgroup` in keyspace/shard discovery Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * fix import Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * fix ineffassign Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * missing import Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * `vtorc`: add stats for discovery workers Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * get count from backend Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * rm unused map Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> --------- Signed-off-by: Manan Gupta <manan@planetscale.com> Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com> Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> Co-authored-by: Manan Gupta <35839558+GuptaManan100@users.noreply.github.com> Co-authored-by: Dirkjan Bussink <d.bussink@gmail.com> * Bp pr 17558 pr 17858.slack19.0 (#615) * VReplication: Improve error handling in VTGate VStreams (vitessio#17558) Signed-off-by: Tom Thornton <thomaswilliamthornton@gmail.com> * Backport vitessio#17858 --------- Signed-off-by: Tom Thornton <thomaswilliamthornton@gmail.com> * `slack-19.0`: re-backport tweaks from vitessio#17911 (#621) * fix bug in reverse `if` Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * simplify Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * add `ReadTabletCountsByShard` test Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * use map of map Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * capitalize Cell Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * gofmt lint Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * fix plural in names Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> --------- Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> --------- Signed-off-by: Malcolm Akinje <makinje@slack-corp.com> Signed-off-by: Manan Gupta <manan@planetscale.com> Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com> Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> Signed-off-by: Tom Thornton <thomaswilliamthornton@gmail.com> Signed-off-by: Malcolm Akinje <malcolm.akinje@gmail.com> Co-authored-by: Tim Vaillancourt <tim@timvaillancourt.com> Co-authored-by: Manan Gupta <35839558+GuptaManan100@users.noreply.github.com> Co-authored-by: Dirkjan Bussink <d.bussink@gmail.com> Co-authored-by: Tom Thornton <thomaswilliamthornton@gmail.com>

…d Journal Events (#585) * VTGate VStream: Ensure reasonable delivery time for reshard journal event (vitessio#16639) Signed-off-by: Malcolm Akinje <malcolm.akinje@gmail.com> Signed-off-by: Malcolm Akinje <makinje@slack-corp.com> * Backport sqlparser patch for v15->v19 upgrade: 14763 Fix accepting bind variables in time related function calls (#590) * Fix accepting bind variables in time related function calls. (vitessio#14763) Signed-off-by: Manan Gupta <manan@planetscale.com> * fix test --------- Signed-off-by: Manan Gupta <manan@planetscale.com> Co-authored-by: Manan Gupta <35839558+GuptaManan100@users.noreply.github.com> * Upgrade vitess addons to 0.19.8 (#591) This upgrade allows us to control whether vtorc raises problems or not via an environment variable. Signed-off-by: Eduardo J. Ortega U. <5791035+ejortegau@users.noreply.github.com> * Use prefix in all vtorc check and recover logs (vitessio#17526) (#592) This is a backport of vitessio#17526 . Original PR description below: Description This is meant to make recovery actions more easily identified from the logs. See vitessio#17465 Signed-off-by: Eduardo J. Ortega U. <5791035+ejortegau@users.noreply.github.com> * `slack-19.0`: various backports for `vtorc`, part 2 (#596) * Ensure all topo read calls consider `--topo_read_concurrency` (vitessio#17276) Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * Revert "add keyrange support for vtorc clusters_to_watch (#457)" This reverts commit 45c2199. * [release-19.0] `vtorc`: require topo for `Healthy: true` in `/debug/health` (vitessio#17129) (vitessio#17351) Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> Signed-off-by: Manan Gupta <manan@planetscale.com> Co-authored-by: vitess-bot[bot] <108069721+vitess-bot[bot]@users.noreply.github.com> Co-authored-by: Tim Vaillancourt <tim@timvaillancourt.com> Co-authored-by: Manan Gupta <manan@planetscale.com> * `vtorc`: fetch all tablets from cells once + filter during refresh (vitessio#17388) Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * Support KeyRange in `--clusters_to_watch` flag (vitessio#17604) Signed-off-by: Manan Gupta <manan@planetscale.com> * missing func Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * Add api end point to print the current database state in VTOrc (vitessio#15485) Signed-off-by: Manan Gupta <manan@planetscale.com> --------- Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> Signed-off-by: Manan Gupta <manan@planetscale.com> Co-authored-by: vitess-bot[bot] <108069721+vitess-bot[bot]@users.noreply.github.com> Co-authored-by: Manan Gupta <manan@planetscale.com> Co-authored-by: Manan Gupta <35839558+GuptaManan100@users.noreply.github.com> * `slack-19.0`: `vtorc`: improve handling of partial cell topo results (#599) * `vtorc`: improve handling of partial cell topo results Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * add unit test Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * improve test Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * add comments Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * move sort to test Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * goimports Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> --------- Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * `slack-19.0`: skip tests that will fail on v15 downgrade testing (#605) Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * `slack-19.0`: Add stats for shards watched by VTOrc (#606) * Add stats for shards watched by VTOrc Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * Use len() in make --------- Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * Add `GetServerStatus` RPC to use in PRS (vitessio#16022) (#607) Signed-off-by: Manan Gupta <manan@planetscale.com> Co-authored-by: Manan Gupta <35839558+GuptaManan100@users.noreply.github.com> * backport/patch connection pool bug/perf fixes (#604) * [release-19.0] smartconnpool: do not allow connections to starve (vitessio#17675) (vitessio#17683) Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com> Co-authored-by: vitess-bot[bot] <108069721+vitess-bot[bot]@users.noreply.github.com> * smartconnpool: Better handling for idle expiration (vitessio#17756) Signed-off-by: Vicent Marti <vmg@strn.cat> --------- Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com> Signed-off-by: Vicent Marti <vmg@strn.cat> Co-authored-by: vitess-bot[bot] <108069721+vitess-bot[bot]@users.noreply.github.com> Co-authored-by: Vicent Martí <42793+vmg@users.noreply.github.com> Co-authored-by: Tim Vaillancourt <tim@timvaillancourt.com> * pool: reopen connection closed by idle timeout (vitessio#17818) (#609) Signed-off-by: Harshit Gangal <harshit@planetscale.com> Signed-off-by: Vicent Martí <42793+vmg@users.noreply.github.com> Co-authored-by: Harshit Gangal <harshit@planetscale.com> Co-authored-by: Vicent Martí <42793+vmg@users.noreply.github.com> * VReplication: Support excluding lagging tablets and use this in vstream manager (vitessio#17835) (#612) * `slack-19.0`: backport v22 VTOrc optimizations, part 2 (#613) * `vtorc`: remove duplicate instance read from backend (vitessio#17834) Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * `vtorc`: add index for `inst.ReadInstanceClusterAttributes` table scan Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> --------- Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * Add stats for shards watched by VTOrc, purge stale shards (vitessio#17815) (#616) * --consolidator-query-waiter-cap to set the max number of waiter for consolidated query (vitessio#17244) (#614) Signed-off-by: Jun Wang <jun.wang@demonware.net> Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> Co-authored-by: jwang <121262788+jwangace@users.noreply.github.com> Co-authored-by: Jun Wang <jun.wang@demonware.net> * `slack-19.0` backport v22 `vtorc` optimizations + stats, part 3 (#618) * Remove unused code in discovery queue creation (vitessio#17515) Signed-off-by: Manan Gupta <manan@planetscale.com> * vtorc: Cleanup unused code (vitessio#15508) Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com> * `vtorc`: cleanup discover queue, add concurrency flag (vitessio#17825) Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * `vtorc`: add tablets watched stats Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * fix missing merge conflict update Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * `vtorc`: skip unnecessary `inst.ReadTablet` in `logic.LockShard(...)` Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * `vtorc`: use `errgroup` in keyspace/shard discovery Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * fix import Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * fix ineffassign Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * missing import Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * `vtorc`: add stats for discovery workers Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * get count from backend Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * rm unused map Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> --------- Signed-off-by: Manan Gupta <manan@planetscale.com> Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com> Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> Co-authored-by: Manan Gupta <35839558+GuptaManan100@users.noreply.github.com> Co-authored-by: Dirkjan Bussink <d.bussink@gmail.com> * Bp pr 17558 pr 17858.slack19.0 (#615) * VReplication: Improve error handling in VTGate VStreams (vitessio#17558) Signed-off-by: Tom Thornton <thomaswilliamthornton@gmail.com> * Backport vitessio#17858 --------- Signed-off-by: Tom Thornton <thomaswilliamthornton@gmail.com> * `slack-19.0`: re-backport tweaks from vitessio#17911 (#621) * fix bug in reverse `if` Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * simplify Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * add `ReadTabletCountsByShard` test Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * use map of map Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * capitalize Cell Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * gofmt lint Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * fix plural in names Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> --------- Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * fix releasing the global read lock when mysqlshell backup fails (vitessio#17000) (#623) Signed-off-by: Renan Rangel <rrangel@slack-corp.com> * VStream API: allow keyspace-level heartbeats to be streamed (vitessio#16593) (#620) * VStream API: allow keyspace-level heartbeats to be streamed (vitessio#16593) Signed-off-by: Malcolm Akinje <makinje@slack-corp.com> * `slack-19.0` backport v22 `vtorc` optimizations + stats, part 3 (#618) * Remove unused code in discovery queue creation (vitessio#17515) Signed-off-by: Manan Gupta <manan@planetscale.com> * vtorc: Cleanup unused code (vitessio#15508) Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com> * `vtorc`: cleanup discover queue, add concurrency flag (vitessio#17825) Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * `vtorc`: add tablets watched stats Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * fix missing merge conflict update Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * `vtorc`: skip unnecessary `inst.ReadTablet` in `logic.LockShard(...)` Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * `vtorc`: use `errgroup` in keyspace/shard discovery Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * fix import Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * fix ineffassign Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * missing import Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * `vtorc`: add stats for discovery workers Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * get count from backend Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * rm unused map Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> --------- Signed-off-by: Manan Gupta <manan@planetscale.com> Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com> Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> Co-authored-by: Manan Gupta <35839558+GuptaManan100@users.noreply.github.com> Co-authored-by: Dirkjan Bussink <d.bussink@gmail.com> * Bp pr 17558 pr 17858.slack19.0 (#615) * VReplication: Improve error handling in VTGate VStreams (vitessio#17558) Signed-off-by: Tom Thornton <thomaswilliamthornton@gmail.com> * Backport vitessio#17858 --------- Signed-off-by: Tom Thornton <thomaswilliamthornton@gmail.com> * `slack-19.0`: re-backport tweaks from vitessio#17911 (#621) * fix bug in reverse `if` Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * simplify Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * add `ReadTabletCountsByShard` test Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * use map of map Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * capitalize Cell Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * gofmt lint Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * fix plural in names Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> --------- Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> --------- Signed-off-by: Malcolm Akinje <makinje@slack-corp.com> Signed-off-by: Manan Gupta <manan@planetscale.com> Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com> Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> Signed-off-by: Tom Thornton <thomaswilliamthornton@gmail.com> Signed-off-by: Malcolm Akinje <malcolm.akinje@gmail.com> Co-authored-by: Tim Vaillancourt <tim@timvaillancourt.com> Co-authored-by: Manan Gupta <35839558+GuptaManan100@users.noreply.github.com> Co-authored-by: Dirkjan Bussink <d.bussink@gmail.com> Co-authored-by: Tom Thornton <thomaswilliamthornton@gmail.com> * Increase health check channel buffer (vitessio#17821) (#625) Signed-off-by: Manan Gupta <manan@planetscale.com> Signed-off-by: Malcolm Akinje <makinje@slack-corp.com> Co-authored-by: Manan Gupta <35839558+GuptaManan100@users.noreply.github.com> * VStream: Allow for automatic resume after Reshard across VStreams (vitessio#15393) (#627) Signed-off-by: Tanjin Xu <tanjin.xu@slack-corp.com> Co-authored-by: Matt Lord <mattalord@gmail.com> --------- Signed-off-by: Malcolm Akinje <malcolm.akinje@gmail.com> Signed-off-by: Malcolm Akinje <makinje@slack-corp.com> Signed-off-by: Manan Gupta <manan@planetscale.com> Signed-off-by: Eduardo J. Ortega U. <5791035+ejortegau@users.noreply.github.com> Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com> Signed-off-by: Vicent Marti <vmg@strn.cat> Signed-off-by: Harshit Gangal <harshit@planetscale.com> Signed-off-by: Vicent Martí <42793+vmg@users.noreply.github.com> Signed-off-by: Jun Wang <jun.wang@demonware.net> Signed-off-by: Tom Thornton <thomaswilliamthornton@gmail.com> Signed-off-by: Renan Rangel <rrangel@slack-corp.com> Signed-off-by: Tanjin Xu <tanjin.xu@slack-corp.com> Co-authored-by: Tanjin Xu <109303790+tanjinx@users.noreply.github.com> Co-authored-by: Manan Gupta <35839558+GuptaManan100@users.noreply.github.com> Co-authored-by: Eduardo J. Ortega U. <5791035+ejortegau@users.noreply.github.com> Co-authored-by: Tim Vaillancourt <tim@timvaillancourt.com> Co-authored-by: vitess-bot[bot] <108069721+vitess-bot[bot]@users.noreply.github.com> Co-authored-by: Manan Gupta <manan@planetscale.com> Co-authored-by: Vicent Martí <42793+vmg@users.noreply.github.com> Co-authored-by: Harshit Gangal <harshit@planetscale.com> Co-authored-by: Tom Thornton <thomaswilliamthornton@gmail.com> Co-authored-by: jwang <121262788+jwangace@users.noreply.github.com> Co-authored-by: Jun Wang <jun.wang@demonware.net> Co-authored-by: Dirkjan Bussink <d.bussink@gmail.com> Co-authored-by: Renan Rangel <rvrangel@users.noreply.github.com> Co-authored-by: Matt Lord <mattalord@gmail.com>

* VReplication: Improve error handling in VTGate VStreams (vitessio#17558) Signed-off-by: Tom Thornton <thomaswilliamthornton@gmail.com> * Backport vitessio#17858 --------- Signed-off-by: Tom Thornton <thomaswilliamthornton@gmail.com>

twthorn requested review from mattlord and rohit-nayak-ps as code owners February 24, 2025 22:56

github-actions bot added this to the v22.0.0 milestone Feb 24, 2025

Add more vstream metrics for vstream manager

f6f37d0

Signed-off-by: twthorn <thomaswilliamthornton@gmail.com>

twthorn force-pushed the vtgate-add-more-vstream-metrics branch from 6359a99 to f6f37d0 Compare February 24, 2025 23:00

twthorn added 2 commits February 25, 2025 11:00

Fix unit tests by resetting metrics

a9c08db

Signed-off-by: twthorn <thomaswilliamthornton@gmail.com>

Prevent go routine leak in tests

ea6caff

Signed-off-by: twthorn <thomaswilliamthornton@gmail.com>

mattlord reviewed Feb 25, 2025

View reviewed changes

Make metrics shard level

02268f7

Signed-off-by: twthorn <thomaswilliamthornton@gmail.com>

twthorn requested a review from mattlord February 26, 2025 18:59

mattlord approved these changes Feb 27, 2025

View reviewed changes

twthorn added 3 commits February 27, 2025 13:33

Only count errors not due to client

bc9b72b

Signed-off-by: twthorn <thomaswilliamthornton@gmail.com>

Use reset instead of adding zero

5998513

Signed-off-by: twthorn <thomaswilliamthornton@gmail.com>

Switch back to add zero

239367b

Signed-off-by: twthorn <thomaswilliamthornton@gmail.com>

twthorn requested a review from mattlord February 27, 2025 20:22

mattlord approved these changes Feb 27, 2025

View reviewed changes

twthorn requested a review from mattlord February 27, 2025 22:46

mattlord removed the NeedsWebsiteDocsUpdate What it says label Feb 27, 2025

mattlord approved these changes Feb 27, 2025

View reviewed changes

deepthi requested review from notfelineit and shlomi-noach February 28, 2025 18:30

twthorn added a commit to twthorn/vitess that referenced this pull request Feb 28, 2025

Backport vitessio#17858

3cc0fec

twthorn added a commit to slackhq/vitess that referenced this pull request Feb 28, 2025

Backport vitessio#17858

18f39e6

notfelineit approved these changes Mar 3, 2025

View reviewed changes

notfelineit merged commit ea9ea39 into vitessio:main Mar 3, 2025
103 of 104 checks passed

BrewTestBot mentioned this pull request Apr 29, 2025

vitess 22.0.0 Homebrew/homebrew-core#221874

Merged

		labels := []string{sgtid.Keyspace, sgtid.Shard, vs.tabletType.String()}
		vs.vsm.vstreamsEndedWithErrors.Add(labels, 0)

Conversation

twthorn commented Feb 24, 2025 • edited by mattlord Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Issue(s)

Checklist

Uh oh!

vitess-bot bot commented Feb 24, 2025

Review Checklist

General

Tests

Documentation

New flags

If a workflow is added or modified:

Backward compatibility

Uh oh!

codecov bot commented Feb 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

mattlord left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mattlord Feb 25, 2025

Choose a reason for hiding this comment

Uh oh!

twthorn Feb 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mattlord Feb 27, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mattlord left a comment

Choose a reason for hiding this comment

Uh oh!

mattlord Feb 27, 2025

Choose a reason for hiding this comment

Uh oh!

twthorn Feb 27, 2025

Choose a reason for hiding this comment

Uh oh!

twthorn Feb 27, 2025

Choose a reason for hiding this comment

Uh oh!

mattlord Feb 27, 2025

Choose a reason for hiding this comment

Uh oh!

twthorn Feb 27, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

mattlord left a comment

Choose a reason for hiding this comment

Uh oh!

twthorn commented Feb 27, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

twthorn commented Feb 24, 2025 •

edited by mattlord

Loading

codecov bot commented Feb 25, 2025 •

edited

Loading

twthorn Feb 26, 2025 •

edited

Loading