Skip to content

[VTAdmin]#11414

Closed
notfelineit wants to merge 38 commits intovitessio:mainfrom
planetscale:frances/remaining-vtadmin-parity-tickets
Closed

[VTAdmin]#11414
notfelineit wants to merge 38 commits intovitessio:mainfrom
planetscale:frances/remaining-vtadmin-parity-tickets

Conversation

@notfelineit
Copy link
Contributor

@notfelineit notfelineit commented Sep 30, 2022 •
edited
Loading

Description

This PR completes "VTAdmin Parity" by implementing:

  • Validate
  • ValidateShard
  • Topology browser

Validate
Validate is added to the clusters list, since validation occurs at the cluster level:
validate

ValidateShard
validateShard

Related Issue(s)

Checklist

  • "Backport me!" label has been added if this change should be backported
  • Tests were added or are not required
  • Documentation was added or is not required

Deployment Notes

@notfelineit
Add validate
1f66483 
Signed-off-by: notfelineit <notfelineit@gmail.com>
@notfelineit
Merge branch 'main' of github.com:vitessio/vitess into frances/remain…
b0e7b53 
…ing-vtadmin-parity-tickets
@vitess-bot
Copy link
Contributor

vitess-bot bot commented Sep 30, 2022

Review Checklist

Hello reviewers! 👋 Please follow this checklist when reviewing this Pull Request.

General

  • Ensure that the Pull Request has a descriptive title.
  • If this is a change that users need to know about, please apply the release notes (needs details) label so that merging is blocked unless the summary release notes document is included.

If a new flag is being introduced:

  • Is it really necessary to add this flag?
  • Flag names should be clear and intuitive (as far as possible)
  • Help text should be descriptive.
  • Flag names should use dashes (-) as word separators rather than underscores (_).

If a workflow is added or modified:

  • Each item in Jobs should be named in order to mark it as required.
  • If the workflow should be required, the maintainer team should be notified.

Bug fixes

  • There should be at least one unit or end-to-end test.
  • The Pull Request description should include a link to an issue that describes the bug.

Non-trivial changes

  • There should be some code comments as to why things are implemented the way they are.

New/Existing features

  • Should be documented, either by modifying the existing documentation or creating new documentation.
  • New features should have a link to a feature request issue or an RFC that documents the use cases, corner cases and test cases.

Backward compatibility

  • Protobuf changes should be wire-compatible.
  • Changes to _vt tables and RPCs need to be backward compatible.
  • vtctl command output order should be stable and awk-able.
  • RPC changes should be compatible with vitess-operator
  • If a flag is removed, then it should also be removed from VTop, if used there.

mattlord and others added 18 commits September 30, 2022 17:14
@mattlord
Move dbconfigs to pflag and remove deprecated flags (vitessio#11336)
c82dd87 
* Migrate dbconfigs to pflag

And remove deprecated flags

Signed-off-by: Matt Lord <mattalord@gmail.com>

* Remove no longer needed function

Signed-off-by: Matt Lord <mattalord@gmail.com>

* Address review comments and test failures

Signed-off-by: Matt Lord <mattalord@gmail.com>

* Revert this bit as it changed behavior and caused some failures

Signed-off-by: Matt Lord <mattalord@gmail.com>

* Fix vttestserver unit tests

Signed-off-by: Matt Lord <mattalord@gmail.com>

* Fix vttestserver unit tests

Signed-off-by: Matt Lord <mattalord@gmail.com>

* Update vttestserver help output

Signed-off-by: Matt Lord <mattalord@gmail.com>

Signed-off-by: Matt Lord <mattalord@gmail.com>
@notfelineit
Add validate to cluster page
a880682 
Signed-off-by: notfelineit <notfelineit@gmail.com>
@mgale
fix: objname was not logged correctly (vitessio#11038)
a7eeeae 
Signed-off-by: Michael Gale <mgale@demonware.net>

Signed-off-by: Michael Gale <mgale@demonware.net>
Co-authored-by: Michael Gale <mgale@demonware.net>
@rsajwani
[cli] [etcd2] Migrate etcd2topo flags to pflags (vitessio#11394)
2e05ab0 
* [cli] [etcd2] Migrate etcd2topo flags to pflags

Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com>

* Fixing duplicate registration

Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com>

Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com>
@rsajwani @GuptaManan100
[cli] [topo] Migrate topo flags to pflags (vitessio#11393)
0f31032 
* [cli] [topo] Migrate topo flags to pflags

Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com>

* fix vtorc test

Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com>

* feat: fix VTOrc test

Signed-off-by: Manan Gupta <manan@planetscale.com>

* feat: remove binaries from the list that don't need the flags

Signed-off-by: Manan Gupta <manan@planetscale.com>

* test: update flag help outputs

Signed-off-by: Manan Gupta <manan@planetscale.com>

* code review

Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com>

* fix import formatting

Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com>

Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com>
Signed-off-by: Manan Gupta <manan@planetscale.com>
Co-authored-by: Manan Gupta <manan@planetscale.com>
@GuptaManan100
Enable VTOrc in backup tests (vitessio#11410)
475a1d4 
* feat: enable vtorc in backup tests

Signed-off-by: Manan Gupta <manan@planetscale.com>

* feat: reorder verifying initial replication to happen first

Signed-off-by: Manan Gupta <manan@planetscale.com>

* test: increase timeout for VTOrc recovery to be successful and improve VTOrc performance

Signed-off-by: Manan Gupta <manan@planetscale.com>

Signed-off-by: Manan Gupta <manan@planetscale.com>
[tools/rowlog] Migrate to pflag (vitessio#11412)
b389dc1 
* Pre-work tidying: tidy imports, remove commented out code

Signed-off-by: Andrew Mason <andrew@planetscale.com>

* [rowlog] Migrate to pflag, add flag for testing resumability

Closes vitessio#11408.

Signed-off-by: Andrew Mason <andrew@planetscale.com>

Signed-off-by: Andrew Mason <andrew@planetscale.com>
@deepthi
flags etc: delete old flags and stats, add deprecation notice to rele…
3f73bdb 
…ase notes (vitessio#11402)

* flags etc: delete old flags and stats, add deprecation notice to release notes

Signed-off-by: deepthi <deepthi@planetscale.com>

* vrepl: fix tests to run SwitchReads with tablet_types flag instead of deprecated tablet_type

Signed-off-by: deepthi <deepthi@planetscale.com>

* pitr: fix tests to run SwitchReads with tablet_types flag instead of deprecated tablet_type

Signed-off-by: deepthi <deepthi@planetscale.com>

Signed-off-by: deepthi <deepthi@planetscale.com>
@GuptaManan100
feat: fix vtbackup binary by adding the flags it needs (vitessio#11417)
c9f018b 
Signed-off-by: Manan Gupta <manan@planetscale.com>

Signed-off-by: Manan Gupta <manan@planetscale.com>
@rohit-nayak-ps
[cli] Topo: Migrate zk2topo and k8stopo to pflag (vitessio#11401)
958d595 
* Migrate zk2topo and k8stopo to pflag

Signed-off-by: Rohit Nayak <rohit@planetscale.com>

* Update flags help output

Signed-off-by: Rohit Nayak <rohit@planetscale.com>

* Fix failing tests

Signed-off-by: Rohit Nayak <rohit@planetscale.com>

* Fix help text output

Signed-off-by: Rohit Nayak <rohit@planetscale.com>

* Fix help output

Signed-off-by: Rohit Nayak <rohit@planetscale.com>

* Fix help text output

Signed-off-by: Rohit Nayak <rohit@planetscale.com>

* Add vtorc to topo binaries

Signed-off-by: Rohit Nayak <rohit@planetscale.com>

* Define k8stopo flags on all topo binaries. Change way flags are defined for tests

Signed-off-by: Rohit Nayak <rohit@planetscale.com>

* Update test help output files

Signed-off-by: Rohit Nayak <rohit@planetscale.com>

Signed-off-by: Rohit Nayak <rohit@planetscale.com>
@frouioui @GuptaManan100
Release notes 14.0.3 on main (vitessio#11406)
fb15da5 
* Release notes for 14.0.3

Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr>

* Addition of the release summary for `v14.0.3` (vitessio#11396)

* addition of empty release summary for 14.0.3

Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr>

* feat: add a line describing the VTOrc bug fix

Signed-off-by: Manan Gupta <manan@planetscale.com>

Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr>
Signed-off-by: Manan Gupta <manan@planetscale.com>
Co-authored-by: Manan Gupta <manan@planetscale.com>

Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr>
Signed-off-by: Manan Gupta <manan@planetscale.com>
Co-authored-by: Manan Gupta <manan@planetscale.com>
@GuptaManan100
VTOrc Standardisation and Cleanup (vitessio#11416)
afda892 
* feat: remove unused package

Signed-off-by: Manan Gupta <manan@planetscale.com>

* feat: replace usage of external test package with require

Signed-off-by: Manan Gupta <manan@planetscale.com>

* feat: replace using external copied dependency with actual dependency

Signed-off-by: Manan Gupta <manan@planetscale.com>

* feat: remove unused external package

Signed-off-by: Manan Gupta <manan@planetscale.com>

Signed-off-by: Manan Gupta <manan@planetscale.com>
@harshit-gangal
test: deflake TestIdleTimeoutCreateFail (vitessio#11411)
c7116c9 
Signed-off-by: Harshit Gangal <harshit@planetscale.com>

Signed-off-by: Harshit Gangal <harshit@planetscale.com>
@systay
fix: do not rewrite single columns in derived tables (vitessio#11419)
4342564 
Signed-off-by: Andres Taylor <andres@planetscale.com>

Signed-off-by: Andres Taylor <andres@planetscale.com>
@maxenglander
expose vtbackup stats at --port /metrics (vitessio#11388)
55695d1 
* expose /metrics from vtbackup http --port

Signed-off-by: Max Englander <max@planetscale.com>

* -mathrand seed

Signed-off-by: Max Englander <max@planetscale.com>

Signed-off-by: Max Englander <max@planetscale.com>
Remove duplicated vt/log import (vitessio#11423)
4a31b91 
Signed-off-by: Andrew Mason <andrew@planetscale.com>

Signed-off-by: Andrew Mason <andrew@planetscale.com>
@GuptaManan100
feat: add a function to get the flag set for a given command (vitessi…
31fb403 
…o#11424)

Signed-off-by: Manan Gupta <manan@planetscale.com>

Signed-off-by: Manan Gupta <manan@planetscale.com>
@notfelineit @deepthi
[cli] [vtctl] Migrate all vtctl commands to pflag (vitessio#11320)
68e5398 
* [cli] [vtctl] Migrate all vtctl commands to `pflag`

Closes vitessio#11304.

Signed-off-by: Andrew Mason <andrew@planetscale.com>

* fix test flags

Signed-off-by: Andrew Mason <andrew@planetscale.com>

* Handle subflag argument parsing without fully moving to cobra (yet!)

Signed-off-by: Andrew Mason <andrew@planetscale.com>

* update examples

Signed-off-by: Andrew Mason <andrew@planetscale.com>

* strip off double-dash

Signed-off-by: Andrew Mason <andrew@planetscale.com>

* Fix pflag getting tripped on shard 0 bounded shard names

Signed-off-by: Andrew Mason <andrew@planetscale.com>

* remove commented-out code

Signed-off-by: Andrew Mason <andrew@planetscale.com>

* update release notes

Signed-off-by: Andrew Mason <andrew@planetscale.com>

* fix ExecuteHook, which takes no flags

Signed-off-by: Andrew Mason <andrew@planetscale.com>

* add deprecation notice around interspersal

Signed-off-by: Andrew Mason <andrew@planetscale.com>

* fix flags for vtctld2 web calls

Signed-off-by: Andrew Mason <andrew@planetscale.com>

* Run make web_build

Signed-off-by: notfelineit <notfelineit@gmail.com>

* test: fix vtctld_web_test

Signed-off-by: deepthi <deepthi@planetscale.com>

* update test assertions

Signed-off-by: Andrew Mason <andrew@planetscale.com>

* just double-dash everything

Signed-off-by: Andrew Mason <andrew@planetscale.com>

* Run make web_build

Signed-off-by: notfelineit <notfelineit@gmail.com>

Signed-off-by: Andrew Mason <andrew@planetscale.com>
Signed-off-by: notfelineit <notfelineit@gmail.com>
Signed-off-by: deepthi <deepthi@planetscale.com>
Co-authored-by: notfelineit <notfelineit@gmail.com>
Co-authored-by: deepthi <deepthi@planetscale.com>
notfelineit and others added 4 commits October 3, 2022 11:15
@notfelineit
Run lint prettier fix
0d50bc2 
Signed-off-by: notfelineit <notfelineit@gmail.com>
@dbussink @deepthi @rsajwani
Move towards MySQL 8.0 as the default template generation (vitessio#1…
e93e197 
…1153)

* Move towards MySQL 8.0 as the default template generation

This upgrades the remaining things to Ubuntu 20.04 and makes MySQL 8.0
the default we run tests against. We still have tests for MySQL 5.7 but
those are now explicitly opted into.

This should finish up the Ubuntu 20.04 upgrade and also makes things
easier for the future when we need to upgrade again.

Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com>

* CI: rename shard vtorc_8.0 to vtorc_5.7, change expected test output for 8.0

Signed-off-by: deepthi <deepthi@planetscale.com>

* CI: increase timeout for 8.0 tests on the actual test step from 30 to 45 mins

Signed-off-by: deepthi <deepthi@planetscale.com>

* CI: increase timeout to 45 minutes for mysql57 tests too. We really only need this for vtorc, but I've made the change to the template so all tests get it.

Signed-off-by: deepthi <deepthi@planetscale.com>

* CI: fix vtorc test to work with both 5.7 and 8.0

Signed-off-by: deepthi <deepthi@planetscale.com>

* CI: missed docker flag in mysql57 template, one more fix to vtorc test

Signed-off-by: deepthi <deepthi@planetscale.com>

* removing spaces from pb file

Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com>

* removing spaces in pb file part 2

Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com>

Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com>
Signed-off-by: deepthi <deepthi@planetscale.com>
Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com>
Co-authored-by: deepthi <deepthi@planetscale.com>
Co-authored-by: Rameez Sajwani <rameezwazirali@hotmail.com>
@notfelineit
Add UI for validate shard
1601ffb 
Signed-off-by: notfelineit <notfelineit@gmail.com>
@mattlord
VDiff: Make restarting VReplication workflow more robust (vitessio#11413
07387b8 
)

* When restarting workflow, use new context

Otherwise the restart will also fail due to the
expired/canceled context:
E0929 14:06:08.618848   85788 vdiff.go:247] Could not restart workflow commerce2customer: rpc error: code = Canceled desc = context canceled, please restart it manually
E0929 14:06:08.619983   85788 vtctl.go:2630] vdiff returning with error: rpc error: code = Canceled desc = context canceled

Signed-off-by: Matt Lord <mattalord@gmail.com>

* Add retry when restarting workflows on ephemeral errors

Signed-off-by: Matt Lord <mattalord@gmail.com>

* Correct mistake that prevented compilation :face-palm:

Signed-off-by: Matt Lord <mattalord@gmail.com>

Signed-off-by: Matt Lord <mattalord@gmail.com>
notfelineit and others added 14 commits October 3, 2022 15:29
@notfelineit
Add validateShard
8b8c655 
Signed-off-by: notfelineit <notfelineit@gmail.com>
@notfelineit
Run make web proto
d18d8a0 
Signed-off-by: notfelineit <notfelineit@gmail.com>
@notfelineit
Refactor validation results and implement validateShard
e26febc 
Signed-off-by: notfelineit <notfelineit@gmail.com>
@rsajwani
Code freeze of release-15.0 (vitessio#11427)
0c257dd 
Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com>
Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr>

Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com>
Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr>
@notfelineit
Add validate
4b153f8 
Signed-off-by: notfelineit <notfelineit@gmail.com>
@notfelineit
Add validate to cluster page
12de204 
Signed-off-by: notfelineit <notfelineit@gmail.com>
@notfelineit
Run lint prettier fix
a812293 
Signed-off-by: notfelineit <notfelineit@gmail.com>
@notfelineit
Add UI for validate shard
8e8b388 
Signed-off-by: notfelineit <notfelineit@gmail.com>
@notfelineit
Add validateShard
23626bb 
Signed-off-by: notfelineit <notfelineit@gmail.com>
@notfelineit
Run make web proto
3057b9c 
Signed-off-by: notfelineit <notfelineit@gmail.com>
@notfelineit
Refactor validation results and implement validateShard
692d452 
Signed-off-by: notfelineit <notfelineit@gmail.com>
@notfelineit
add http shards
49e766d 
Signed-off-by: notfelineit <notfelineit@gmail.com>
@notfelineit
Add GetFullStatus
f4363c5 
Signed-off-by: notfelineit <notfelineit@gmail.com>
@notfelineit
Merge branch 'frances/remaining-vtadmin-parity-tickets' of github.com…
83e2c1a 
…:planetscale/vitess into frances/remaining-vtadmin-parity-tickets

Signed-off-by: notfelineit <notfelineit@gmail.com>
@notfelineit
Copy link
Contributor Author

notfelineit commented Oct 4, 2022

Closing this for #11438

@notfelineit notfelineit closed this Oct 4, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@harshit-gangal harshit-gangal Awaiting requested review from harshit-gangal harshit-gangal will be requested when the pull request is marked ready for review harshit-gangal is a code owner

@mattlord mattlord Awaiting requested review from mattlord mattlord will be requested when the pull request is marked ready for review mattlord is a code owner

@beingnoble03 beingnoble03 Awaiting requested review from beingnoble03 beingnoble03 will be requested when the pull request is marked ready for review beingnoble03 is a code owner

@frouioui frouioui Awaiting requested review from frouioui frouioui will be requested when the pull request is marked ready for review frouioui is a code owner

@rohit-nayak-ps rohit-nayak-ps Awaiting requested review from rohit-nayak-ps rohit-nayak-ps will be requested when the pull request is marked ready for review rohit-nayak-ps is a code owner

@shlomi-noach shlomi-noach Awaiting requested review from shlomi-noach shlomi-noach will be requested when the pull request is marked ready for review shlomi-noach is a code owner

@timvaillancourt timvaillancourt Awaiting requested review from timvaillancourt timvaillancourt will be requested when the pull request is marked ready for review timvaillancourt is a code owner

@systay systay Awaiting requested review from systay systay will be requested when the pull request is marked ready for review systay is a code owner

@arthurschreiber arthurschreiber Awaiting requested review from arthurschreiber arthurschreiber will be requested when the pull request is marked ready for review arthurschreiber is a code owner

@dbussink dbussink Awaiting requested review from dbussink dbussink will be requested when the pull request is marked ready for review dbussink is a code owner

Copilot code review Copilot Awaiting requested review from Copilot Copilot will automatically review once the pull request is marked ready for review

Assignees

No one assigned

Labels

Component: VTAdmin VTadmin interface Component: web UI Type: Feature

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.