Extend join prefilter optimizer to use hash for filter#23858
Merged
feilong-liu merged 1 commit intoprestodb:masterfrom Oct 22, 2024
Merged
Extend join prefilter optimizer to use hash for filter#23858feilong-liu merged 1 commit intoprestodb:masterfrom
feilong-liu merged 1 commit intoprestodb:masterfrom
Conversation
jaystarshot
previously approved these changes
Oct 18, 2024
| * - InnerJoin | ||
| * leftKey = rightKey | ||
| * - scan l | ||
| * - semijoin |
Member
There was a problem hiding this comment.
nit: Can show the hash project here to explain
Contributor
Author
There was a problem hiding this comment.
Updated to cover all three cases
- one join key (not varchar type)
- one join key (varchar type)
- multiple join keys
kaikalur
previously approved these changes
Oct 19, 2024
| List<VariableReferenceExpression> rightKeyList = equiJoinClause.stream().map(EquiJoinClause::getRight).collect(toImmutableList()); | ||
| checkState(IntStream.range(0, leftKeyList.size()).boxed().allMatch(i -> leftKeyList.get(i).getType().equals(rightKeyList.get(i).getType()))); | ||
|
|
||
| boolean hashJoinKey = leftKeyList.size() > 1 || isWideColumn(leftKeyList.get(0)); |
Contributor
There was a problem hiding this comment.
Keep it simple and just do it always for varchar/char type?
6afbd9c
87f824b to
6afbd9c
Compare
jaystarshot
previously approved these changes
Oct 21, 2024
6afbd9c to
f254ee0
Compare
kaikalur
approved these changes
Oct 22, 2024
Contributor
Author
|
@jaystarshot I have updated the code to address the test failure, can you take another look? Thanks! |
jaystarshot
approved these changes
Oct 22, 2024
25 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
JoinPrefilter optimizer was added in #22667 which shows great improvement for a bunch of queries within Meta. However there are two limits of the current implementation
To solve the above limits, this PR:
We can apply the hash as this pre filter is an opportunistic and does not need to be comprehensive.
Motivation and Context
To apply the join prefilter optimizer to more query shapes
Impact
Apply join prefilter to more query shapes
Test Plan
Add to existing plans
Also run verifier suites
This optimizer is default to be disabled as it's not always beneficial, we are considering using HBO to enable it in the future
Contributor checklist
Release Notes
Please follow release notes guidelines and fill in the release notes below.