Skip to content

sqlparser: Improve interval parsing#13165

Merged
dbussink merged 3 commits intovitessio:mainfrom
dbussink:improve-interval-parsing
May 29, 2023
Merged

sqlparser: Improve interval parsing#13165
dbussink merged 3 commits intovitessio:mainfrom
dbussink:improve-interval-parsing

Conversation

@dbussink
Copy link
Copy Markdown
Member

@dbussink dbussink commented May 26, 2023

Parse only strict valid units for intervals. We should not allow any arbitrary string value here. We already had the internal enumerable available, but we were not using yet.

Related Issue(s)

In support of #9647 to then make it easier to add support for functions using intervals.

Checklist

  • "Backport to:" labels have been added if this change should be back-ported
  • Tests were added or are not required
  • Did the new or modified tests pass consistently locally and on the CI
  • Documentation was added or is not required

@vitess-bot vitess-bot bot added NeedsDescriptionUpdate The description is not clear or comprehensive enough, and needs work NeedsIssue A linked issue is missing for this Pull Request NeedsWebsiteDocsUpdate What it says labels May 26, 2023
@vitess-bot
Copy link
Copy Markdown
Contributor

vitess-bot bot commented May 26, 2023 •
edited by GuptaManan100
Loading

Review Checklist

Hello reviewers! 👋 Please follow this checklist when reviewing this Pull Request.

General

  • Ensure that the Pull Request has a descriptive title.
  • If this is a change that users need to know about, please apply the release notes (needs details) label so that merging is blocked unless the summary release notes document is included.
  • If a test is added or modified, there should be a documentation on top of the test to explain what the expected behavior is what the test does.

If a new flag is being introduced:

  • Is it really necessary to add this flag?
  • Flag names should be clear and intuitive (as far as possible)
  • Help text should be descriptive.
  • Flag names should use dashes (-) as word separators rather than underscores (_).

If a workflow is added or modified:

  • Each item in Jobs should be named in order to mark it as required.
  • If the workflow should be required, the maintainer team should be notified.

Bug fixes

  • There should be at least one unit or end-to-end test.
  • The Pull Request description should include a link to an issue that describes the bug.

Non-trivial changes

  • There should be some code comments as to why things are implemented the way they are.

New/Existing features

  • Should be documented, either by modifying the existing documentation or creating new documentation.
  • New features should have a link to a feature request issue or an RFC that documents the use cases, corner cases and test cases.

Backward compatibility

  • Protobuf changes should be wire-compatible.
  • Changes to _vt tables and RPCs need to be backward compatible.
  • vtctl command output order should be stable and awk-able.
  • RPC changes should be compatible with vitess-operator
  • If a flag is removed, then it should also be removed from VTop, if used there.

@dbussink dbussink requested a review from vmg May 26, 2023 13:34
@dbussink dbussink removed NeedsDescriptionUpdate The description is not clear or comprehensive enough, and needs work NeedsWebsiteDocsUpdate What it says NeedsIssue A linked issue is missing for this Pull Request labels May 26, 2023
@github-actions github-actions bot added this to the v17.0.0 milestone May 26, 2023
@dbussink dbussink requested a review from frouioui as a code owner May 26, 2023 14:42
@dbussink dbussink force-pushed the improve-interval-parsing branch from 9b475bb to c57ede6 Compare May 26, 2023 15:40
@dbussink
Copy link
Copy Markdown
Member Author

dbussink commented May 26, 2023

@systay Looks like there's an upgrade / downgrade failing test unrelated to the changes here:

=== RUN   TestOrderBy
    orderby_test.go:70: 
        	Error Trace:	/home/runner/work/vitess/vitess/go/test/endtoend/vtgate/queries/orderby/cmp.go:195
        	            				/home/runner/work/vitess/vitess/go/test/endtoend/vtgate/queries/orderby/cmp.go:65
        	            				/home/runner/work/vitess/vitess/go/test/endtoend/vtgate/queries/orderby/orderby_test.go:70
        	Error:      	Received unexpected error:
        	            	VT13001: [BUG] in scatter query: complex ORDER BY expression: reverse(id2) (errno 1815) (sqlstate HY000) during query: select /*vt+ PLANNER=Gen4 */ id1, id2 from t4 order by reverse(id2) desc
        	Test:       	TestOrderBy
        	Messages:   	[Vitess Error] for query: select /*vt+ PLANNER=Gen4 */ id1, id2 from t4 order by reverse(id2) desc
--- FAIL: TestOrderBy (0.04s)
FAIL

dbussink added 3 commits May 29, 2023 13:50
@dbussink
sqlparser: Improve interval parsing
fe6fca5 
Parse only strict valid units for intervals. We should not allow any
arbitrary string value here. We already had the internal enumerable
available, but we were not using yet.

Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com>
@dbussink
sqlparser: Explicit parsing for date functions with intervals
0d9384d 
In order to better be able to implement these in the evalengine, it's
easier to parse them explicitly in the parser. This ensures we have the
interval type etc. directly available instead of having to infer it
from a separate expression.

Also makes parsing these functions more strict and better match MySQL.

Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com>
@dbussink
sqlparser: Remove INTERVAL as general expression
9c2a16f 
INTERVAL is not a possible general expression, but only used in specific
cases. With the previous change, we already have it all handled for date
functions. The only remaining cases are for + and - where this is
handled.

Those cases are a bit special. You can have an interval on both sides of
a +, but for a - it can only be done on the right hand side.

Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com>
@dbussink dbussink force-pushed the improve-interval-parsing branch from e5bd5d6 to 9c2a16f Compare May 29, 2023 11:51
GuptaManan100
GuptaManan100 approved these changes May 29, 2023
View reviewed changes
Copy link
Copy Markdown
Contributor

@GuptaManan100 GuptaManan100 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@GuptaManan100
Copy link
Copy Markdown
Contributor

GuptaManan100 commented May 29, 2023

The upgrade downgrade failure is unrelated and being fixed by #13183

systay
systay reviewed May 29, 2023
View reviewed changes
go/test/endtoend/vtgate/queries/misc/misc_test.go
mcmp.Exec("SET time_zone = '+00:00'")
mcmp.AssertMatches("select '2020-01-01' + interval month(DATE_SUB(FROM_UNIXTIME(1234), interval 1 month))-1 month", `[[CHAR("2020-12-01")]]`)
mcmp.AssertMatches("select DATE_ADD(MIN(FROM_UNIXTIME(1673444922)),interval -DAYOFWEEK(MIN(FROM_UNIXTIME(1673444922)))+1 DAY)", `[[DATETIME("2023-01-08 13:48:42")]]`)
mcmp.AssertMatches("select '2020-01-01' + interval month(date_sub(FROM_UNIXTIME(1234), interval 1 month))-1 month", `[[CHAR("2020-12-01")]]`)
Copy link
Copy Markdown
Collaborator

@systay systay May 29, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: why change this? the capitalized form is also valid, right?

Copy link
Copy Markdown
Member Author

@dbussink dbussink May 29, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@systay It's because we normalize explicit function names here to lowercase for explicitly known parser implementations and I didn't want to diverge from that here. Similarly here, for example month also was emitted as lowercase.

It's also how MySQL itself normalizes it if you use it for example in a default in create table if you then issue a show create table statement. We try to follow that as the canonical form then.

Copy link
Copy Markdown
Collaborator

@systay systay May 29, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

here you are changing the query we are sending down to vtgate/mysql. why did that need to change? we should follow the canonical form for the printing of ASTs, but we have to be able to receive the queries in any valid form, right?

Copy link
Copy Markdown
Member Author

@dbussink dbussink May 29, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah yeah, it would also still be valid. I was pretty aggressive though with the find / replace to lowercase it which is why this one was caught as well 😄.

systay
systay approved these changes May 29, 2023
View reviewed changes
Copy link
Copy Markdown
Collaborator

@systay systay left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dbussink dbussink merged commit 2ba1edc into vitessio:main May 29, 2023
@dbussink dbussink deleted the improve-interval-parsing branch May 29, 2023 13:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@systay systay systay approved these changes

@harshit-gangal harshit-gangal Awaiting requested review from harshit-gangal harshit-gangal is a code owner

@vmg vmg Awaiting requested review from vmg

@frouioui frouioui Awaiting requested review from frouioui frouioui is a code owner

+1 more reviewer

@GuptaManan100 GuptaManan100 GuptaManan100 approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

Component: Query Serving Type: Internal Cleanup

Projects

None yet

Milestone

v17.0.0

Development

Successfully merging this pull request may close these issues.

3 participants

@dbussink @GuptaManan100 @systay