Skip to content

Filter by keyspace earlier in tabletgateways WaitForTablets(...)#15347

Merged
deepthi merged 10 commits intovitessio:mainfrom
timvaillancourt:srvtopo-FindAllTargets-keyspaces_to_watch
Feb 27, 2024
Merged

Filter by keyspace earlier in tabletgateways WaitForTablets(...)#15347
deepthi merged 10 commits intovitessio:mainfrom
timvaillancourt:srvtopo-FindAllTargets-keyspaces_to_watch

Conversation

@timvaillancourt
Copy link
Contributor

@timvaillancourt timvaillancourt commented Feb 24, 2024

Description

This PR causes the tabletgateway .WaitForTablets(...) method to filter targets by keyspace (set by --keyspaces_to_watch) earlier in the process

Currently, the logic under .WaitForTablets(...):

  1. Uses srvtopo.FindAllTargets(...) to fetch the targets by-cell for any keyspace (even if --keyspaces_to_watch is set)
    • This calls .GetSrvKeyspace(...) on the toposerver for every keyspace in goroutines w/wait group
  2. Later targets that do not match --keyspaces_to_watch are removed from targets []*querypb.Target

Instead this PR implements the filtering in srvtopo.FindAllTargets(...) by not-fetching keyspaces from the topo that will just get filtered anyways. This can avoid topo calls and unnecessary filtering of results on deployments with many keyspaces and a subset of keyspaces set as --keyspaces_to_watch

The changed codepath and signatures are only called by vtgate/tabletgateway so this change had a limited impact

Related Issue(s)

Checklist

  • "Backport to:" labels have been added if this change should be back-ported to release branches
  • If this change is to be back-ported to previous releases, a justification is included in the PR description
  • Tests were added or are not required
  • Did the new or modified tests pass consistently locally and on CI?
  • Documentation was added or is not required

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>
@vitess-bot
Copy link
Contributor

vitess-bot bot commented Feb 24, 2024

Review Checklist

Hello reviewers! 👋 Please follow this checklist when reviewing this Pull Request.

General

  • Ensure that the Pull Request has a descriptive title.
  • Ensure there is a link to an issue (except for internal cleanup and flaky test fixes), new features should have an RFC that documents use cases and test cases.

Tests

  • Bug fixes should have at least one unit or end-to-end test, enhancement and new features should have a sufficient number of tests.

Documentation

  • Apply the release notes (needs details) label if users need to know about this change.
  • New features should be documented.
  • There should be some code comments as to why things are implemented the way they are.
  • There should be a comment at the top of each new or modified test to explain what the test does.

New flags

  • Is this flag really necessary?
  • Flag names must be clear and intuitive, use dashes (-), and have a clear help text.

If a workflow is added or modified:

  • Each item in Jobs should be named in order to mark it as required.
  • If the workflow needs to be marked as required, the maintainer team must be notified.

Backward compatibility

  • Protobuf changes should be wire-compatible.
  • Changes to _vt tables and RPCs need to be backward compatible.
  • RPC changes should be compatible with vitess-operator
  • If a flag is removed, then it should also be removed from vitess-operator and arewefastyet, if used there.
  • vtctl command output order should be stable and awk-able.

@vitess-bot vitess-bot bot added NeedsBackportReason If backport labels have been applied to a PR, a justification is required NeedsDescriptionUpdate The description is not clear or comprehensive enough, and needs work NeedsIssue A linked issue is missing for this Pull Request NeedsWebsiteDocsUpdate What it says labels Feb 24, 2024
@github-actions github-actions bot added this to the v20.0.0 milestone Feb 24, 2024
Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>
Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>
Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>
@codecov
Copy link

codecov bot commented Feb 24, 2024

Codecov Report

Attention: Patch coverage is 87.50000% with 1 lines in your changes are missing coverage. Please review.

Project coverage is 65.43%. Comparing base (d8f771c) to head (612d8c4).
Report is 9 commits behind head on main.

Files Patch % Lines
go/vt/srvtopo/discover.go 85.71% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #15347      +/-   ##
==========================================
+ Coverage   65.39%   65.43%   +0.03%     
==========================================
  Files        1561     1561              
  Lines      193524   193630     +106     
==========================================
+ Hits       126563   126696     +133     
+ Misses      66961    66934      -27     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>
Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>
Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>
Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>
Copy link
Member

@mattlord mattlord left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! I only had a few very minor comments/nits. Let me know what you think and I can come back to this quickly and help to get it merged.

Thanks, @timvaillancourt !

@mattlord mattlord added Type: Enhancement Logical improvement (somewhere between a bug and feature) Component: Query Serving and removed NeedsBackportReason If backport labels have been applied to a PR, a justification is required NeedsDescriptionUpdate The description is not clear or comprehensive enough, and needs work NeedsWebsiteDocsUpdate What it says labels Feb 27, 2024
@mattlord
Copy link
Member

@timvaillancourt do you mind creating a real quick issue as well for this? 🙏

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>
Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>
@mattlord mattlord self-requested a review February 27, 2024 17:48
@timvaillancourt
Copy link
Contributor Author

LGTM! I only had a few very minor comments/nits. Let me know what you think and I can come back to this quickly and help to get it merged.

@mattlord no problem, I try to imitate the style of the project I'm committing to so this is good knowledge 👍

An issue is created here: #15373

@mattlord mattlord removed the NeedsIssue A linked issue is missing for this Pull Request label Feb 27, 2024
Copy link
Member

@mattlord mattlord left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, @timvaillancourt ! ❤️

Copy link
Collaborator

@deepthi deepthi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work! I'm loving these little optimizations that y'all have been doing.

@deepthi deepthi merged commit 236f84c into vitessio:main Feb 27, 2024
@timvaillancourt timvaillancourt deleted the srvtopo-FindAllTargets-keyspaces_to_watch branch February 27, 2024 19:50
timvaillancourt added a commit to slackhq/vitess that referenced this pull request Feb 27, 2024
…itessio#15347)

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>
timvaillancourt added a commit to slackhq/vitess that referenced this pull request Feb 29, 2024
…itessio#15347)

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>
timvaillancourt added a commit to slackhq/vitess that referenced this pull request Mar 12, 2024
…itessio#15347) (#229)

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>
timvaillancourt added a commit to slackhq/vitess that referenced this pull request Mar 15, 2024
…izations (#227)

* Load `--grpc_auth_static_client_creds` file once (vitessio#15030)

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>

* Filter by keyspace earlier in `tabletgateway`s `WaitForTablets(...)` (vitessio#15347)

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>

* Limit concurrent creation of healthcheck gRPC connections (vitessio#15053)

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>

* go mod tidy

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>

* Update MySQL apt package and GPG signature (vitessio#14785)

Signed-off-by: Matt Lord <mattalord@gmail.com>

* remove unrelated workflow files from v20

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>

---------

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>
Signed-off-by: Matt Lord <mattalord@gmail.com>
Co-authored-by: Matt Lord <mattalord@gmail.com>
timvaillancourt added a commit to slackhq/vitess that referenced this pull request May 30, 2024
…itessio#15347)

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>
timvaillancourt added a commit to slackhq/vitess that referenced this pull request Jul 9, 2024
* Make `Durabler` interface methods public (vitessio#15548)

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>
Signed-off-by: Manan Gupta <manan@planetscale.com>
Co-authored-by: Manan Gupta <manan@planetscale.com>

* Load `--grpc_auth_static_client_creds` file once (vitessio#15030)

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>

* Limit concurrent creation of healthcheck gRPC connections (vitessio#15053)

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>

* Filter by keyspace earlier in `tabletgateway`s `WaitForTablets(...)` (vitessio#15347)

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>

* Use slack-15.0 as previous release

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>

* empty commit

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>

* force ci to run

* Update GH Action runners

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>

* test templates

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>

* set GH access token in build

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>

* Fix reparent old tests

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>

* Remove CIs we don't need

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>

* Remove CIs we don't need again

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>

* Add private repo setup to upgrade_downgrade_test_backups_e2e.yml

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>

* Add private repo setup to more CI

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>

* remove CI skip logic for upstream stuff

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>

* CODEOWNERS

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>

* [release-19.0] Add timeout to all the contexts used for RPC calls in vtorc (vitessio#15991) (vitessio#16103)

Signed-off-by: Manan Gupta <manan@planetscale.com>

* `slack-vitess-r15.0.5`: forward-port consul topo limits PR #111 (#297)

* `slack-vitess-r14.0.5`: allow conn overrides in consul topo (#111)

* `slack-vitess-r14.0.5`: allow conn overrides in consul topo

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>

* fix e2e test

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>

---------

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>

* Update flags tests that didn't exist in v14

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>

---------

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>

* update vtcombo e2e

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>

* Fix err with installing percona-xtrabackup-24

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>

* `slack-vitess-r15.0.5`: fix races in `Unit Test (Race)` CI, fix "old" reparent CIs (#356)

* update vtcombo e2e test

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>

* Fix bad merge conflict fix

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>

* go mod tidy

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>

* update vtcombo e2e test again

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>

* [release-19.0] Upgrade the Golang version to `go1.22.5` (vitessio#16322)

Signed-off-by: GitHub <noreply@github.com>
Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr>
Co-authored-by: frouioui <frouioui@users.noreply.github.com>
Co-authored-by: Florent Poinsard <florent.poinsard@outlook.fr>

* merge conflict fixes

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>

* make vtadmin_web_proto_types

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>

---------

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>
Signed-off-by: Manan Gupta <manan@planetscale.com>
Signed-off-by: GitHub <noreply@github.com>
Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr>
Co-authored-by: Manan Gupta <manan@planetscale.com>
Co-authored-by: Manan Gupta <35839558+GuptaManan100@users.noreply.github.com>
Co-authored-by: vitess-bot <139342327+vitess-bot@users.noreply.github.com>
Co-authored-by: frouioui <frouioui@users.noreply.github.com>
Co-authored-by: Florent Poinsard <florent.poinsard@outlook.fr>
arthurschreiber pushed a commit to github/vitess-gh that referenced this pull request Nov 8, 2024
…itessio#15347)

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>
arthurschreiber pushed a commit to github/vitess-gh that referenced this pull request Nov 19, 2024
…itessio#15347)

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Component: Query Serving Type: Enhancement Logical improvement (somewhere between a bug and feature)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Feature Request: reduce topo calls in tabletgateway's WaitForTablets(...)

3 participants