Skip to content

Move horizon planning to operators#11622

Merged
systay merged 8 commits intovitessio:mainfrom
planetscale:operator-horizon
Nov 4, 2022
Merged

Move horizon planning to operators#11622
systay merged 8 commits intovitessio:mainfrom
planetscale:operator-horizon

Conversation

@frouioui
Copy link
Copy Markdown
Member

@frouioui frouioui commented Nov 2, 2022 •
edited by systay
Loading

Description

A refactoring that introduces a Horizon as a logical operator.

During query planning, we start by figuring out where we are going to send the queries. This is done by inspecting the FROM & WHERE clauses.

A horizon is what we call in the planner the stuff that is outside the FROM ... WHERE clause. If we are able to send the whole query to a single shard, we don't do much more planning - we simply add the rest of the clauses (projections, ordering, limit, aggregation and more) to the query and we are done.

If we can't delegate everything to MySQL, we do more planning, and this is what we call horizon planning.

This change moves the horizon into the operator tree, instead of having special handling of derived tables, unions and subqueries.

Related Issue(s)

#11626

Checklist

  • "Backport me!" label has been added if this change should be backported
  • Tests were added or are not required
  • Documentation was added or is not required

Deployment Notes

systay added 7 commits November 1, 2022 14:33
@systay
move the horizon to live on the operator tree instead of outside it
d0486b9 
Signed-off-by: Andres Taylor <andres@planetscale.com>
@systay
remove select statements from UNION
d67328d 
Signed-off-by: Andres Taylor <andres@planetscale.com>
@systay
move code around
7248cee 
Signed-off-by: Andres Taylor <andres@planetscale.com>
@systay
move all the different operator steps into a single method
25d7a36 
Signed-off-by: Andres Taylor <andres@planetscale.com>
@systay
move logic around
5a150e2 
Signed-off-by: Andres Taylor <andres@planetscale.com>
@systay
make methods private
6cc2f7d 
Signed-off-by: Andres Taylor <andres@planetscale.com>
@systay
a little refactoring and infrastructure to do horizon planning on the…
ad5000a 
… operators

Signed-off-by: Andres Taylor <andres@planetscale.com>
@vitess-bot
Copy link
Copy Markdown
Contributor

vitess-bot bot commented Nov 2, 2022 •
edited
Loading

Review Checklist

Hello reviewers! 👋 Please follow this checklist when reviewing this Pull Request.

General

  • Ensure that the Pull Request has a descriptive title.
  • If this is a change that users need to know about, please apply the release notes (needs details) label so that merging is blocked unless the summary release notes document is included.

If a new flag is being introduced:

  • Is it really necessary to add this flag?
  • Flag names should be clear and intuitive (as far as possible)
  • Help text should be descriptive.
  • Flag names should use dashes (-) as word separators rather than underscores (_).

If a workflow is added or modified:

  • Each item in Jobs should be named in order to mark it as required.
  • If the workflow should be required, the maintainer team should be notified.

Bug fixes

  • There should be at least one unit or end-to-end test.
  • The Pull Request description should include a link to an issue that describes the bug.

Non-trivial changes

  • There should be some code comments as to why things are implemented the way they are.

New/Existing features

  • Should be documented, either by modifying the existing documentation or creating new documentation.
  • New features should have a link to a feature request issue or an RFC that documents the use cases, corner cases and test cases.

Backward compatibility

  • Protobuf changes should be wire-compatible.
  • Changes to _vt tables and RPCs need to be backward compatible.
  • vtctl command output order should be stable and awk-able.
  • RPC changes should be compatible with vitess-operator
  • If a flag is removed, then it should also be removed from VTop, if used there.

@frouioui frouioui added Type: Enhancement Logical improvement (somewhere between a bug and feature) Component: Query Serving labels Nov 2, 2022
frouioui
frouioui commented Nov 2, 2022
View reviewed changes
go/vt/vtgate/planbuilder/operators/apply_join.go
Comment on lines 70 to +71
// Clone implements the Operator interface
func (a *ApplyJoin) Clone(inputs []Operator) Operator {
func (a *ApplyJoin) clone(inputs []Operator) Operator {
Copy link
Copy Markdown
Member Author

@frouioui frouioui Nov 2, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: we can also change the comment Clone -> clone

@systay
update operator tests
079bc33 
Signed-off-by: Andres Taylor <andres@planetscale.com>
frouioui
frouioui commented Nov 3, 2022
View reviewed changes
go/vt/vtgate/planbuilder/operators/horizon_planning.go
Comment on lines +52 to +53
}
func planSingleShardRoute(statement sqlparser.SelectStatement, rb *Route, horizon *Horizon) (Operator, error) {
Copy link
Copy Markdown
Member Author

@frouioui frouioui Nov 3, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: new line :)

frouioui
frouioui commented Nov 3, 2022
View reviewed changes
go/vt/vtgate/planbuilder/operators/horizon.go
"vitess.io/vitess/go/vt/vtgate/planbuilder/plancontext"
)

// Horizon is an operator we use until we decide how to handle the source to the horizon.
Copy link
Copy Markdown
Member Author

@frouioui frouioui Nov 3, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure I fully understand this. We add the Horizon operator into the tree until we have decided how the rest of the tree will do horizon planning?

Copy link
Copy Markdown
Member Author

@frouioui frouioui Nov 3, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking at the operator test data, it feels like it encapsulates each node in the tree.

Copy link
Copy Markdown
Collaborator

@systay systay Nov 3, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. We need to do the physical route planning before we do the horizon planning

frouioui
frouioui commented Nov 3, 2022
View reviewed changes
go/vt/vtgate/planbuilder/operators/operator.go
Comment on lines +17 to +20
// Package operators contains the operators used to plan queries.
/*
The operators go through a few phases while planning:
1. Logical
Copy link
Copy Markdown
Member Author

@frouioui frouioui Nov 3, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's a nice write up

Copy link
Copy Markdown
Collaborator

@systay systay Nov 3, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll add info about horizons in the op tree as well

frouioui
frouioui commented Nov 3, 2022
View reviewed changes
Copy link
Copy Markdown
Member Author

@frouioui frouioui left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, can't approve as this is my PR.

@frouioui
Copy link
Copy Markdown
Member Author

frouioui commented Nov 3, 2022

We should update the Pull Request description.

@systay systay marked this pull request as ready for review November 3, 2022 14:38
GuptaManan100
GuptaManan100 approved these changes Nov 4, 2022
View reviewed changes
Copy link
Copy Markdown
Contributor

@GuptaManan100 GuptaManan100 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like the changes!

@systay systay merged commit f541e96 into vitessio:main Nov 4, 2022
@systay systay deleted the operator-horizon branch November 4, 2022 09:59
@systay systay mentioned this pull request Mar 29, 2023
Merged
@frouioui frouioui mentioned this pull request Mar 30, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@systay systay systay left review comments

@harshit-gangal harshit-gangal Awaiting requested review from harshit-gangal harshit-gangal is a code owner

+1 more reviewer

@GuptaManan100 GuptaManan100 GuptaManan100 approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

Component: Query Serving Type: Enhancement Logical improvement (somewhere between a bug and feature)

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@frouioui @systay @GuptaManan100