Skip to content

Schema Tracking Refactor: Merge schema-tracking in health-streamer into schema.Engine#13121

Merged
GuptaManan100 merged 42 commits intovitessio:mainfrom
planetscale:get-schema-rpc
May 31, 2023
Merged

Schema Tracking Refactor: Merge schema-tracking in health-streamer into schema.Engine#13121
GuptaManan100 merged 42 commits intovitessio:mainfrom
planetscale:get-schema-rpc

Conversation

@GuptaManan100
Copy link
Contributor

@GuptaManan100 GuptaManan100 commented May 19, 2023 •
edited
Loading

Description

This PR refactors schema tracking to merge the schema tracking in healthstreamer into schema.Engine.

We still have the schema tracking in healthstreamer around because of backward compatibility, which we will remove in a later release. However, we have removed the ticker on which we would run the reload function and instead do it in response to a signal from the schema.Engine.

The schema tracking introduced in schema.Engine stores a little more information than what we had in healthstreamer to support the upcoming use case of #12967.

This PR also fixes the issue described in #13182. Now, we all the health streamer to write to the database only when the tablet is in a primary serving state. So, when the call to tm.QueryServiceControl.SetServingType succeeds, we can now guarantee that the health streamer will not be writing anything anymore.

Website docs - vitessio/website#1487

Related Issue(s)

Checklist

  • "Backport to:" labels have been added if this change should be back-ported
  • Tests were added or are not required
  • Did the new or modified tests pass consistently locally and on the CI
  • Documentation was added or is not required - SchemaTracker refactor documentation website#1487

Deployment Notes

@GuptaManan100
test: add test to verify the expectations of health stream
4e75fe9 
Signed-off-by: Manan Gupta <manan@planetscale.com>
@GuptaManan100
test: augment the test to also test the case where views are enabled …
122c5e6 
…in vttablets

Signed-off-by: Manan Gupta <manan@planetscale.com>
@GuptaManan100
feat: use the schema engine notifier to run the reload in health-stre…
714a975 
…amer

Signed-off-by: Manan Gupta <manan@planetscale.com>
@GuptaManan100
feat: deprecate the flag controlling the interval of health-streamer …
36758f2 
…reload

Signed-off-by: Manan Gupta <manan@planetscale.com>
@GuptaManan100
test: augment test to verify schema tracking works for alters in view…
9923f76 
…s as well

Signed-off-by: Manan Gupta <manan@planetscale.com>
@GuptaManan100
feat: add a view table type and augment loadTable function to assign …
a950d15 
…it correctly

Signed-off-by: Manan Gupta <manan@planetscale.com>
@GuptaManan100
feat: change health streamer to use the tables got from the schemaEng…
69c3554 
…ine instead of detecting the schema changes on its own

Signed-off-by: Manan Gupta <manan@planetscale.com>
@vitess-bot vitess-bot bot added NeedsDescriptionUpdate The description is not clear or comprehensive enough, and needs work NeedsIssue A linked issue is missing for this Pull Request NeedsWebsiteDocsUpdate What it says labels May 19, 2023
@vitess-bot
Copy link
Contributor

vitess-bot bot commented May 19, 2023

Review Checklist

Hello reviewers! 👋 Please follow this checklist when reviewing this Pull Request.

General

  • Ensure that the Pull Request has a descriptive title.
  • If this is a change that users need to know about, please apply the release notes (needs details) label so that merging is blocked unless the summary release notes document is included.
  • If a test is added or modified, there should be a documentation on top of the test to explain what the expected behavior is what the test does.

If a new flag is being introduced:

  • Is it really necessary to add this flag?
  • Flag names should be clear and intuitive (as far as possible)
  • Help text should be descriptive.
  • Flag names should use dashes (-) as word separators rather than underscores (_).

If a workflow is added or modified:

  • Each item in Jobs should be named in order to mark it as required.
  • If the workflow should be required, the maintainer team should be notified.

Bug fixes

  • There should be at least one unit or end-to-end test.
  • The Pull Request description should include a link to an issue that describes the bug.

Non-trivial changes

  • There should be some code comments as to why things are implemented the way they are.

New/Existing features

  • Should be documented, either by modifying the existing documentation or creating new documentation.
  • New features should have a link to a feature request issue or an RFC that documents the use cases, corner cases and test cases.

Backward compatibility

  • Protobuf changes should be wire-compatible.
  • Changes to _vt tables and RPCs need to be backward compatible.
  • vtctl command output order should be stable and awk-able.
  • RPC changes should be compatible with vitess-operator
  • If a flag is removed, then it should also be removed from VTop, if used there.

@github-actions github-actions bot added this to the v17.0.0 milestone May 19, 2023
@GuptaManan100
feat: make the schema engine capable of storing the table and view in…
9a8202e 
…formation in a database

Signed-off-by: Manan Gupta <manan@planetscale.com>
@GuptaManan100
feat: remove unused functions
b4a7f95 
Signed-off-by: Manan Gupta <manan@planetscale.com>
harshit-gangal
harshit-gangal reviewed May 23, 2023
View reviewed changes
harshit-gangal
harshit-gangal reviewed May 23, 2023
View reviewed changes
@GuptaManan100
feat: fix health streamer tests
5643738 
Signed-off-by: Manan Gupta <manan@planetscale.com>
@GuptaManan100 GuptaManan100 added Type: Enhancement Logical improvement (somewhere between a bug and feature) Component: Query Serving labels May 23, 2023
@GuptaManan100
test: fix tests by also deleting the tablet record for the tablets th…
90e0b56 
…at are removed

Signed-off-by: Manan Gupta <manan@planetscale.com>
harshit-gangal
harshit-gangal reviewed May 23, 2023
View reviewed changes
go/test/endtoend/cluster/cluster_process.go

// StreamTabletHealthUntil invokes a HealthStream on a local cluster Vttablet and
// returns the responses. It waits until a certain condition is met. The amount of time to wait is an input that it takes.
func (cluster *LocalProcessCluster) StreamTabletHealthUntil(ctx context.Context, vttablet *Vttablet, timeout time.Duration, condition func(shr *querypb.StreamHealthResponse) bool) error {
Copy link
Member

@harshit-gangal harshit-gangal May 23, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there are existing usages of StreamTabletHealth can those also use this utility function?

Copy link
Contributor Author

@GuptaManan100 GuptaManan100 May 25, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is an inherent difference between the two. StreamTabletHealth gets you specified number of stream responses back, which can then be processed. On the other hand StreamTabletHealthUntil is meant to be used where the test doesn't care if we consume more responses and only verifies if a certain check is ever true.
The tests currently using StreamTabletHealth actually can't really use StreamTabletHealthUntil because they want to run a test on the next packet, and not an eventual packet.

harshit-gangal
harshit-gangal reviewed May 23, 2023
View reviewed changes
go/test/endtoend/cluster/vtctldclient_process.go
Comment on lines +97 to +103
func (vtctldclient *VtctldClientProcess) PlannedReparentShard(Keyspace string, Shard string, alias string) (err error) {
output, err := vtctldclient.ExecuteCommandWithOutput(
"PlannedReparentShard",
fmt.Sprintf("%s/%s", Keyspace, Shard),
"--new-primary", alias)
if err != nil {
log.Errorf("error in PlannedReparentShard output %s, err %s", output, err.Error())
Copy link
Member

@harshit-gangal harshit-gangal May 23, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there are other places as well we are doing this call, like InitializeShard can it be reused

Copy link
Contributor Author

@GuptaManan100 GuptaManan100 May 25, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could, but eventually we want to move to using vtctldclient and not vtctlclient. So I thought it prudent to add a PRS call to vtctldclient. This is what we should be using going forward in all our tests.

harshit-gangal
harshit-gangal reviewed May 23, 2023
View reviewed changes
harshit-gangal
harshit-gangal reviewed May 23, 2023
View reviewed changes
harshit-gangal
harshit-gangal reviewed May 23, 2023
View reviewed changes
harshit-gangal
harshit-gangal reviewed May 23, 2023
View reviewed changes
harshit-gangal
harshit-gangal reviewed May 23, 2023
View reviewed changes
harshit-gangal
harshit-gangal reviewed May 23, 2023
View reviewed changes
@GuptaManan100 GuptaManan100 mentioned this pull request May 24, 2023
Closed
@GuptaManan100
test: fix engine test flakiness
f7b496e 
Signed-off-by: Manan Gupta <manan@planetscale.com>
@GuptaManan100
Merge remote-tracking branch 'upstream/main' into get-schema-rpc
c8d891e 
Signed-off-by: Manan Gupta <manan@planetscale.com>
@GuptaManan100
test: add health-streamer test
96cec2c 
Signed-off-by: Manan Gupta <manan@planetscale.com>
@GuptaManan100
feat: fix data race in fakedb
f778e0b 
Signed-off-by: Manan Gupta <manan@planetscale.com>
This was referenced May 29, 2023
Merged
Closed
@GuptaManan100
Merge remote-tracking branch 'upstream/main' into get-schema-rpc
ff01bf9 
Signed-off-by: Manan Gupta <manan@planetscale.com>
@GuptaManan100
Merge remote-tracking branch 'upstream/main' into get-schema-rpc
dcff98a 
Signed-off-by: Manan Gupta <manan@planetscale.com>
@GuptaManan100
test: add header comments to tests
993fa6b 
Signed-off-by: Manan Gupta <manan@planetscale.com>
@GuptaManan100
feat: rename views table columns to table_schema and table_name
05d3546 
Signed-off-by: Manan Gupta <manan@planetscale.com>
@GuptaManan100
feat: fix detectViewChange query
ee7562d 
Signed-off-by: Manan Gupta <manan@planetscale.com>
harshit-gangal
harshit-gangal reviewed May 30, 2023
View reviewed changes
Copy link
Member

@harshit-gangal harshit-gangal left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good overall. this is a significant improvement on the schema engine and more cleaner code and also fixes the issue with view change tracking.

frouioui
frouioui reviewed May 30, 2023
View reviewed changes
frouioui
frouioui reviewed May 30, 2023
View reviewed changes
@GuptaManan100
feat: fix the summary
e00653f 
Signed-off-by: Manan Gupta <manan@planetscale.com>
@GuptaManan100 GuptaManan100 mentioned this pull request May 30, 2023
Merged
@GuptaManan100 GuptaManan100 removed the NeedsWebsiteDocsUpdate What it says label May 30, 2023
@GuptaManan100 GuptaManan100 merged commit 9e892eb into vitessio:main May 31, 2023
@GuptaManan100 GuptaManan100 deleted the get-schema-rpc branch May 31, 2023 11:15
@GuptaManan100 GuptaManan100 mentioned this pull request May 31, 2023
Merged
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@harshit-gangal harshit-gangal harshit-gangal approved these changes

@frouioui frouioui frouioui approved these changes

@mattlord mattlord Awaiting requested review from mattlord mattlord is a code owner

@rohit-nayak-ps rohit-nayak-ps Awaiting requested review from rohit-nayak-ps rohit-nayak-ps is a code owner

@derekperkins derekperkins Awaiting requested review from derekperkins derekperkins is a code owner

@systay systay Awaiting requested review from systay systay is a code owner

@shlomi-noach shlomi-noach Awaiting requested review from shlomi-noach shlomi-noach is a code owner

@deepthi deepthi Awaiting requested review from deepthi

Assignees

No one assigned

Labels

Component: Query Serving Type: Enhancement Logical improvement (somewhere between a bug and feature)

Projects

None yet

Milestone

v17.0.0

Development

Successfully merging this pull request may close these issues.

Bug Report: Errant gtid possible in schema tracking in health streamer

3 participants

@GuptaManan100 @harshit-gangal @frouioui