[feat](Nereids) support nereids hint position detaction #39113
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Thank you for your contribution to Apache Doris. Since 2024-03-18, the Document has been moved to doris-website. |
|
run buildall |
TPC-H: Total hot run time: 40231 ms |
|
run buildall |
TPC-H: Total hot run time: 40374 ms |
TPC-DS: Total hot run time: 209400 ms |
ClickBench: Total hot run time: 31.57 s |
morrySnow
reviewed
Aug 9, 2024
| @@ -72,6 +72,19 @@ lexer grammar DorisLexer; | |||
| public void markUnclosedComment() { | |||
| has_unclosed_bracketed_comment = true; | |||
| } | |||
|
|
|||
| // This variable will hold the external state | |||
| private boolean channel2; | |||
There was a problem hiding this comment.
why need this var? i think we should always let hint go to channel 2 ?
There was a problem hiding this comment.
when i let channel 2 always open, it would have problem when parsing because in channel lexer would add some char before original string to let parser ignore it. For example: it would make "/* leading(t1 t2) */" into "[@-1,15:36 = '/leading(t1 t2) */', xxx" which when i want to parse it, i would meet eof first then parse exit. So it make a switch to control this behavior
| @@ -56,6 +56,10 @@ | |||
|
|
|||
| /**LogicalPlanBuilderForCreateView*/ | |||
| public class LogicalPlanBuilderForCreateView extends LogicalPlanBuilder { | |||
| public LogicalPlanBuilderForCreateView() { | |||
| super(null); | |||
There was a problem hiding this comment.
i think it should use hint too? for example we could use hint to change sql mode, if create view cannot process it correctly it could generate wrong column type. or cannot analyze create view statement correctly.
| @@ -49,6 +49,10 @@ | |||
| public class LogicalPlanBuilderForSyncMv extends LogicalPlanBuilder { | |||
| private Optional<String> querySql; | |||
|
|
|||
| public LogicalPlanBuilderForSyncMv() { | |||
| super(null); | |||
Comment on lines
+299
to
+301
| DorisLexer hintLexer = new DorisLexer(new CaseInsensitiveStream(CharStreams.fromString(sql))); | ||
| hintLexer.setChannel2(true); | ||
| CommonTokenStream hintTokenStream = new CommonTokenStream(hintLexer); |
There was a problem hiding this comment.
antlr could generate one stream and get specific channel token from it by getHiddenTokensToRight, so could we use it to get hint in visitRegularQuerySpecification?
There was a problem hiding this comment.
getHiddenTokensToRight does not work in this case, I tried to get it but it failed. I try to use List hintTokens = hintTokenStream.getHiddenTokensToRight(0, 2); but it failed to get token list out from channel2
Comment on lines
+312
to
+314
| CommonTokenStream newHintTokenStream = new CommonTokenStream(newHintLexer); | ||
| DorisParser hintParser = new DorisParser(newHintTokenStream); | ||
| ParserRuleContext hintContext = parseFunction.apply(hintParser); |
There was a problem hiding this comment.
we could write a new parser to process hint seperately?
There was a problem hiding this comment.
I think it is no need to write a new parser but use exist parser logic as hint syntax, the only thing we need to care about is position and new hint type, position is solved by selectHintMap we added, new hint type is filtered by original hint parser. So it is no needed to add a new parser with same logic, we can only use channel 2 to deal with hint seperately
| @@ -36,6 +36,10 @@ | |||
| */ | |||
| public class PLSqlLogicalPlanBuilder extends LogicalPlanBuilder { | |||
|
|
|||
| public PLSqlLogicalPlanBuilder() { | |||
| super(null); | |||
|
run buildall |
TPC-H: Total hot run time: 40108 ms |
TPC-DS: Total hot run time: 207614 ms |
ClickBench: Total hot run time: 31.14 s |
englefly
previously approved these changes
Aug 12, 2024
github-actions
bot
added
the
approved
|
PR approved by at least one committer and no changes requested. |
|
PR approved by anyone and no changes requested. |
|
run buildall |
github-actions
bot
removed
the
approved
TPC-H: Total hot run time: 40782 ms |
TPC-DS: Total hot run time: 196162 ms |
ClickBench: Total hot run time: 31.34 s |
LiBinfeng-01
force-pushed
the
support_hint_position
branch
from
234725c to
2e14ee4
Compare
|
run buildall |
TPC-H: Total hot run time: 40220 ms |
|
PR approved by at least one committer and no changes requested. |
morrySnow
approved these changes
Aug 15, 2024
LiBinfeng-01
added a commit
to LiBinfeng-01/doris
that referenced
this pull request
Aug 15, 2024
When use hint in wrong position or use unsupport hint, use channel(2) to filter it out
LiBinfeng-01
added a commit
to LiBinfeng-01/doris
that referenced
this pull request
Aug 15, 2024
When use hint in wrong position or use unsupport hint, use channel(2) to filter it out
LiBinfeng-01
added a commit
to LiBinfeng-01/doris
that referenced
this pull request
Aug 15, 2024
When use hint in wrong position or use unsupport hint, use channel(2) to filter it out
LiBinfeng-01
added a commit
to LiBinfeng-01/doris
that referenced
this pull request
Aug 15, 2024
When use hint in wrong position or use unsupport hint, use channel(2) to filter it out
morrySnow
pushed a commit
that referenced
this pull request
Aug 16, 2024
924060929
added a commit
to 924060929/incubator-doris
that referenced
this pull request
Aug 20, 2024
LiBinfeng-01
added a commit
to LiBinfeng-01/doris
that referenced
this pull request
Aug 23, 2024
When use hint in wrong position or use unsupport hint, use channel(2) to filter it out
LiBinfeng-01
added a commit
to LiBinfeng-01/doris
that referenced
this pull request
Aug 23, 2024
…duplicate (apache#39590) nereids timeout cause of parser cost too long time intro by： apache#39113 ## Proposed changes Issue Number: close #xxx <!--Describe your changes.-->
yiguolei
pushed a commit
that referenced
this pull request
Aug 24, 2024
dataroaring
pushed a commit
that referenced
this pull request
Sep 19, 2024
When use hint in wrong position or use unsupport hint, use channel(2) to filter it out
LiBinfeng-01
added a commit
to LiBinfeng-01/doris
that referenced
this pull request
Sep 20, 2024
…duplicate (apache#39590) nereids timeout cause of parser cost too long time intro by： apache#39113 ## Proposed changes Issue Number: close #xxx <!--Describe your changes.-->
dataroaring
pushed a commit
that referenced
this pull request
Sep 20, 2024
mongo360
pushed a commit
to mongo360/doris
that referenced
this pull request
Dec 11, 2024
apache#39416) cherry-pick from master apache#39113 When use hint in wrong position or use unsupport hint, use channel(2) to filter it out
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When use hint in wrong position or use unsupport hint, use channel(2) to filter it out